CAMBRIDGE LANGUAGE TEACHING LIBRARY 
A series of authoritative books on subjects of central importance for 
all language teachers 



In this series: 

Teaching and Learning Languages by Earl W. Steuick 

Communicating Naturally in a Second Language - Theory and practice 
in language teaching by Wilga M. Risers 

Speaking in Many Tongues - Essays in foreign language teaching 
by Wilga M. Rivers 

Teaching the Spoken Language - An approach based on the analysis of 
conversational English by Gillian Brown and George Yule 

A Foundation Course for Language Teachers by Tom McArthur 

Foreign and Second Language Learning - Language-acquisition 
research and its implications for the classroom by William Littlewood 

Communicative Methodology in Language Teaching - The roles of 
fluency and accuracy by Christopher Brumfit 

The Context of Language Teaching by Jack C. Richards 

English for Science and Technology - A discourse approach 
by Louis Trimble 

Approaches and Methods in Language Teaching - A description and 
analysis by Jack C. Richards and Theodore S. Rodgers 

Images and Options ''n the Language Classroom by Earl W. Stevick 

Culture Bound - Bridging the cultural gap in language teaching 
edited by Joyce Merrill V aides 

Interactive Language Teaching edited by Wilga M. Rivers 

Designing Tasks for the Communicative Classroom by David Nunan 

Second Language Teacher Education edited by Jack C. Richards and 
David Nunan 

The Language Teaching Matrix by Jack C. Richards 

Discourse Analysis for Language Teachers by Michael McCarthy 

Discourse and Language Education by Evelyn Hatch 

Collaborative Language Learning and Teaching edited by David Nunan 

Research Methods in Language Learning by David Nunan 



Research Methods in 
Language Learning 



David Nunan 

National Centre for English Language Teaching and Research 
Macquarie University 



la Cambridge 

^.■W UNIVERSITY PRESS 



Piihlislicd by the Prow Syndicate of the University of Cambridge 
Tlic I'irt Buildin(;, I ruinpington Street, Cambridge CB2 IRP 
40 West 20th Street, New York, NY 1001 1-421 1, USA 
10 Stamft)rd Road, Oaklcigh, VictorLi 3 166, Australia 

& Cambridge University Press 1992 

hirst published 1992 

Printed in the United States of America 

Library of Congress Calaloging-in-Piiblication Data 
Nunan, David. 

Research methixis in language learning / David Nunan. 

p. cm. - (C'.imbritlge langungc teaching library) 

Includes bibliogr.iphical references and index. 

ISHN I) 52 1 -419.17-9 (hardback) - ISBN 0-521-42968-4 (papcrKick) 

I . l.aiigii.igc ami l.inguagcs - Study and teaching - Research - 

Methodology. I. Title. II. .Series. 

P5.1.N87 1992 

418'.(X)7 - dc20 91-44022 

CIP 

A catalog record for this h(K)k is available from the British Library 



ISBN 0 52 141 9.17-9 hardback 
ISBN 0-521-42968-4 paficrback 



To my colleagues in the School of English and Linguistics and the National 
Centre for English Language Teaching and Research, Macquarie 
University, this book is affectionately dedicated. 



Contents 



Preface xi 

1 < An introduction to research methods and traditions 1 

Research traditions in applied linguistics 1 
I The status of knowledge 10 
Some key concepts in research 12 
Action research 17 
Conclusion 20 
Questions and tasks 21 
Further reading 23 

' 2 ' The experimental method 24 

The context of exp»erimentation 24 

The logic of statistical inference 28 
\ Additional statistical tools 37 
' Types of experiments 40 

The psychometric study: an example 41 

Conclusion 47 

Questions and tasks 48 

Further reading 51 



3 , Ethnography 52 

Principles of ethnographic research 53 

The reliability and validity of ethnography 58 

The importance of context in ethnographic inquiry 64 

Contrasting psychometry and ethnography 68 

Conclusion 71 

Questions and tasks 71 

Further reading 73 



4 Case study 74 

Defining case studies 74 

Reliability and validity of case study research 79 
Single case research 81 

vi 



Contents 



The case study: an example 83 
Conclusion 88 
Questions and tasks 89 
Further reading 90 

5 Classroom observation and research 91 

Methods of classroom observation and research 91 

A review of research 102 

Classroom research: sample studies 105 

Conclusion 109 

Questions and tasks 1 10 

Further reading 114 

6 Introspective methods 115 

Some early introspective methods 1 16 

Diary studies 118 

Retrospection 124 

A sample study using introspection 129 

Conclusion 132 

Questions and tasks 132 

Further reading 135 

7 Elicitation techniques 136 

Production tasks 136 
Surveys 140 
Questionnaires 143 
Interviews 149 
A sample study 153 
Conclusion 156 
Questions and tasks 157 
Further reading 158 

8 Interaction analysis 159 

Comparing discourse analysis, interaction analysis, and conversation 

analysis 159 

Child-adult interaction 162 

Adult-adult interaction: interpersonal encounters 168 
Adult-adult interaction: transactional encounters 170 
Interaction analysis in a cross-cultural context 174 
Ci)nclusi(m 177 
Questions and tasks 178 
Further reading 183 



9 Program evaluation 184 

Defining evaluation 184 
The relationship between assessment and evaluation 
The importance of process data in program evaluation 
Points of focus for program evaluation 190 
Evaluation and research 193 
Elements in the design of an evaluation study 196 
Doing evaluation: a case study 201 
Conclusion 208 
Questions and tasks 209 
Further reading 210 

— 

10 1 Doing research 211 

Developing a research question 
The literature revievvr 216 
Implementing the research projea 
Presenting the research 220 
Conclusion 226 
Questions and tasks 227 
Further reading 228 

Glossary of key terms in research 229 

References 234 



211 
218 



Index 245 



I 



Preface 



Over the last few years, two plieiiomena of major significance for tliis lx)ok 
have emerged. The first of these is the strengthening of a research orientation 
to language learning and teaching. The second is a broadening of the research 
ctucrprise to embrace the collaborative involvement of teachers themselves in 
research. 

Within the language teaching literature there are numerous works con- 
taining, at worst, wish lists for teacher action and, at best, powerful rhetorical 
prescriptions for practice. In both cases, the precepts tend to be couched in 
the form of received wisdom - in other words, exhortations for one line of 
action rather than another are argued logico-deductively rather than on the 
basis of empirical evidence about what teachers and learners actually do, 
inside and outside the classroom, as they teach, learn, and use language. 

Over the last ten years, this picture has begun to change, the change itselt 
prompted, at least in part, by practitioners who have grown tired of the 
swings and roundabouts of pedagogic fashion. While position papers and log- 
ico-deductive argumentation have not disappeared from the scene (and I am 
not suggesting for a moment that they should), they are counterbalanced by 
empirical approaches to inquiry. I believe that these days, when confronted 
by pedagogical questions and problems, researchers and teachers arc more 
likely than was the case ten or fifteen years ago to seek relevant data, either 
through their own research, or through the research of others. Research activ- 
ity has increased to the point where those who favour logico-deductive solu- 
tions to pedagogic problems are beginning to argue that there is too nuich 
research. 

If teachers are to benefit from the research of others, and if they are to con- 
textualise research outcomes against the reality of their own classrooms, they 
need to be able to read the research reports of others in an informed and crit- 
ical way. Unfortunately, published research is all too often presented in neat, 
unproblematic packages, and critical skills are needed to get beneath the sur- 
face and evaluate the reliability and validity of research outcomes. A major 
function of this book, in addition to providing a contemporary account of 
the 'what' and the 'how' of research, is to help nonresearchers develop the 
critical, analytical skills which will enable them to read and evaluate research 
reports in an informed and knowledgeable way. 

Two alternative conceptions of the nature of research provide a point of 
tension within the bookl The first view is that external truths exist 'out there' 
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somewhere. According to this view, the function of research is to uncover 
these truths. The second view is that truth is a negotiable commodity contin- 
gent upon the historical context within which phenomena are observed and 
interpreted. Further, research 'standards are subject to change in the light of 
practice [which] would seem to indicate that the search for a substantive uni- 
versal, ahistorical methodology is futile' (Chalmers 1990; 21). 

While I shall strive to provide a balanced introduction to these alternative 
traditions, 1 must declare myself at the outset for the second. Accordingly, in 
the book I shall urge the reader to exercise caution in applying research out- 
comes derived in one context to other contexts removed in time and space. 

This second, 'context-bound' attitude to research entails a rather different 
role for the classroom practitioner than the first. If knowledge is tentative and 
contingent upon context, rather than absolute, then I believe that practitio- 
ners, rather than being consumers of other people's research, should adopt a 
research orientation to their own ciassroomsJ There is evidence that the 
teacher-researcher movement is alive and well and gathering strength. How- 
ever, if the momentum which has gathered is not to falter, and if the teacher- 
researcher movement is not to become yet another fad, then significant num- 
bers of teachers, graduate students, and others will need skills in planning, 
implementing, and evaluating research. Accordingly, a second aim of this 
book is to assist the reader to develop relevant research skills. At the end of 
the book, readers should be able to formulate realistic research questions, 
adopt appropriate procedures for collecting and analysing data, and present 
the fruits of their research in a form accessible to others. 

1 should like to thank all those individuals who assisted in the development 
of the ideas in this book. While these researchers, teachers, learners, and grad- 
uate students are too numerous to mention, 1 trust that they will recognise 
the contributions which they have made. One person who deserves explicit 
acknowledgment is Geoff Brindley, who provided many useful references and 
who helped to synthesise the ideas set out in Chapter 7. Thanks are also due 
to the anonymous reviewers, whose thoughtful and detailed comments were 
enormously helpful. Finally, grateful thanks go to Ellen Shaw from Cam- 
bridge University Press, who provided criticism and encouragement in appro- 
priate measure and at just the right time. Thanks also to Suzette Andre, and 
especially to Sandy Graham, who is quite simply the best editor any author 
could wish for. Needless to say, such shortcomings as remain are mine alone. 



1 An introduction to research methods and 
traditions 



Scientists should not be ashamed to admit . . . that hypotheses apfjear in their minds 
along uncharted byways of thought; that they are imaginative and inspirational in 
character; that they are indeed adventures of the mind. 

(Peter Medawar, 1963, "Is the Scientific Paper a Fraud?" BBC Presentation) 

This book is essentially practical in nature. It is intended as an introduction 
to research methods in applied linguistics, and does not assume s[>ecialist 
knowledge of the field. It is written in order to help you to develop a range 
of skills, but more particularly to discuss and critique a wide range of research 
methods, including formal experiments and quasi-experiments; elicitation 
instruments; interviews and questionnaires; observation instruments and 
schedules; introspective methods, including diaries, logs, journals, protocol 
analysis, and stimulated recall; interaction and transcript analysis; ethnog- 
raphy and case studies. Having read the book, you should have a detailed 
appreciation of the basic principles of research design, and you should be able 
to read and critique published studies in applied linguistics. In relation to your 
own teaching, you should be better able to develop strategies for formulating 
questions, and for collecting and analysing data relating to those questions. 

The purpose of this initial chapter is to introduce you to research methods 
and traditions in applied linguistics. The chapter sets the scene for the rest of 
the book, and highlights the central themes underpinning the book. This 
chapter deals with the following questions: 

- What is the difference between quantitative and qualitative research? 

- What do we mean by "the status of knowledge', and why is this of partic- 
ular significance to an understanding of research traditions? 

- What is meant by the terms reliability and validity, and why are they con- 
sidered important in research? 

- What is action research? 



Research traditions in applied linguistics 

1 he very term research is a pejorative one to many practitioners, conjuring 
up images of white-coated scientists plying their arcane trade in laboratories 
filled with mysterious equipment. While research, and the conduct of 



research, involves rigour and the apphcation of specialist knowledge and 
skills, this rather forbidding image is certainly not one I wish to present here. 

I recently asked a group of graduate students who were just beginning a 
research methods course to complete the following statements: 'Research is 
..." and 'Research is carried out in order to ..." Here are some of their 
responses. 

Research is: 

- about inquiry. It has two components: process and product. The process is 
about an area of inquiry and how it is pursued. The product is the knowl- 
edge generated from the process as well as the initial area to be presented. 

- a process which involves (a) defining a problem, (b) stating an objective, and 
(c) formulating an hypothesis. It involves gathering information, classifi- 
cation, analysis, and interpretation to see to what extent the initial objec- 
tive has been achieved. 

- undertaking structured investigation which hopefully results in greater 
understanding of the chosen interest area. Ultimately, this investigation 
becomes accessible to the 'public'. 

- an activity which analyses and critically evaluates some problem. 

- to collect and analyse the data in a specific field with the purpose of proving 
your theory. 

- evaluation, asking questions, investigations, analysis, confirming hypoth- 
eses, overview, gathering and analysing data in a specific field according to 
certain predetermined methods. 

Research is carried out in order to: 

- get a result with scientific methods objectively, not subjectively. 

- solve problems, verify the application of theories, and lead on to new 
insights. 

- enlighten both researcher and any interested readers. 

- prove/disprove new or existing ideas, to characterise phenomena (i.e., the 
language characteristics of a particular population), and to achieve per- 
sonal and community aims. That is, to satisfy the individual's quest but 
also to improve community welfare. 

- prove or disprove, demystify, carry out what is phnned, to support the 
point 6f view, to uncover what is not known, satisfy inquiry. To discover 
the cause of a problem, to find the solution to a problem, etc. 

Certain key terms commonly associated with research appear in these char- 
acterisations. These include: inquiry, knowledge, hypothesis, information, 
classification, analysis, interpretation, structured investigation, understand- 
ing, problem, prove, theory, evaluation, asking questions, analysing data, sci- 
entific method, insight, prove/disprove, characterise phenomena, demystify, 
uncover, satisfy inquiry, solution. The terms, taken together, suggest that 
research is a process of formulating questions, problems, or hypotheses; col- 



2 



leering data or evidence relevant to these questions/problems/ liypotlu-sos; 
and analysing or interpreting these data. The minimal definition to whicli 1 
shall adhere in tliese pages is that research is a systematic process of inquiry 
consisting of three elements or components: (1) a question, problem, or 
hypothesis, (2) data, (3) analysis and inrcrprct.uioii of data. Any .icri\ iry 
which lacks one of these elements (for example, data) I shall classify as some- 
thing other than research. (A short definition of key terms printed in italic 
can be found in the glossary at the end of the b(X)k.) 

Traditionally, writers on research traditions have made a binary ihstinc- 
tion between qualitative and quantitative research, although more recently it 
has been argued that the distinction is simplistic and naive. Reicliardt and 
Cook (cited in Chaudron 1988), for example, argue that in practical terms, 
qualitative and quantitative research are in many respects inilistinguishablc, 
and that 'researchers in no way follow theprinciplesof a supposed paradigm 
without simultaneously assuming methods and values of the alternative par- 
adigms' (Reichardt and Cook 1979: 232). Those who draw a distinction sug- 
gest that quantitative research is obtrusive and controlled, objective, gener- 
alisable, outcome oriented, and assumes the existence of 'facts' which arc 
somehow external to and independent of the observer or researcher. Quali- 
tative research, on the other hand, assumes that all knowledge is relative, that 
there is a subjective element to all knowledge and research, and tiiat holistic, 
ungeneratisable studies are justifiable (an ungeneralisable study is one in 
which the insights and outcomes generated by the research cannot be applied 
to contexts or situations beyond those in which the data were collecteil). In 
metaphorical terms, quantitative research is 'hard' while qualitative research 
is 'soft'. Terms (sometimes used in approbation, sometimes as abuse) com- 
monly associated with the two paradigms are set out in Figure 1.1. 

Ji) an attempt to go beyond the binary distinction between qualitative and 
quantitative research, Chaudron (1988) argues that there are four research 
traditions in applied linguistics. These are the psychometric tradition, inter- 
action analysis, discourse analysis, and ethnography. Typically, lisychomct rif 
investigations seek to determine language gains from different methods and 
materials through the use of the 'experimental method' (to be dealt witii in 
detail in Chapter 2). Interaction analysis in classroom settings investigates 
such relationships as the extent to which learner behaviour is a fimction of 
teacher-determined interaction, and utilises various observation systems and 
schedules for coding classroom interactions. D;sco«rstfi7;M/y5/s analyses class- 
room discourse in linguistic terms through the study of classroom transcripts 
which typically assign utterances to predetermined categories. Finally, eth- 
nography seeks to obtain insights into the classroom as a cultural system 
through naturalistic, 'uncontrolled' observation and description (we shall 
deal with ethnography in Chapter 3). While Chaudron 's aim of attempting 
to transcend the traditional binary distinction is a worthy one, it could be 
argued that discourse analysis and interaction analysis are methods of data 
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Research methods hi language learnhig 



Qualitative research 

Advcx:ates use of qualitative methods 

Concerned with understanding human 

behaviour from the actor's own 

frame of reference 
Naturalistic and uncontrolled 

observation 
Subjective 

Close to the data; the 'insider' 

perpsective 
Grounded, discovery-oriented, 

exploratory, expansionist, 

descriptive, and inductive 

Process-oriented 
Valid: 'real', 'rich', and 'deep' data 
Ungeneralisable: single case studies 
Assumes a dynamic reality 



Quantitative research 

Advocates use of quantitative methods 

Seeks facts or causes of social 
phenomena without regard to the 
subjective states of the individuals 

Obtrusive and controlled measurement 

Objective 

Removed from the data: the 'outsider' 
perspective 

Ungrounded, verification-oriented, 
confirmatory, reductionist, 
inferential, and hypothetical- 
deductive 

Outcome-orien ted 

Reliable; 'hard' and replicable data 

Generalisable; multiple case studies 

Assumes a stable reality 



figure 1. 1 Terms commonly associated with quantitative and qualitative 
approaches to research (adapted from Reichardt and Cook 1979) 

collection rather than distinct research traditions in their own right. In fact 
these methods can be (and have been) utilised by researchers working in both 
the psychometric and ethnographic traditions. For example, ethnographers 
cnn use interaction analysis checklists to supplement their naturalistic obser- 
vations, wliilc psychometric rcsc.irch can use similar schemes to identify and 
measure tlistinctions between different classrooms, teaching methods, 
approaches, and teachers (the studies reported by Spada 1990 are excellent 
examples of such research). 

Grotjahn (1987) provides an insightful analysis of research traditions in 
applied linguistics. He argues that the qualitative-quantitative distinction is 
an oversimplification and that, in analysing actual research studies, it is nec- 
essary to take into consitleration the method of data collection (whether the 
data have Ix'en collectetl exfx.'rinientally or non-expcrimentally); the type of 
data yielded by the investigation (qualitative or quantitative); and the type of 
analysis conducted on the data (whether statistical or interpretive). Mixing 
and matching these variables provides us with two 'pure' research paradigms. 
Paradigm 1 is the 'exploratory-interpretive' one which utilises a non-experi- 
mental method, yields qualitative data, and provides an interpretive analysis 
of that data. The second, or 'analytical-nomological' paradigm, is one in 
which the data are collected through an experiment, and yields quantitative 
data whicii arc subjected to statistical analysis. In addition to these 'pure' 
forms, there are six 'mixed' paradigms which mix and match the three vari- 
ables in different ways. For example, there is an 'experimental-qualitative- 
intcrpretivc' paradigm which utilises an experiment but yields qualitative 
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data, which are analysed interpretively. The different research paradigms 
resulting from mixing and matching these variables are set out in Figure 1.2. 
(It should be pointed out that, while all of these various 'hybrid' forms are 
theoretically possible, some are of extremely unlikely occurrence. For exam- 
ple, it would be unusual for a researcher to go to the trouble of setting 
up a formal experiment yielding quantitative data which are analysed 
interpretively.) 

While I accept Grotjahn's assertion that in the execution of research the 
qualitative-quantitative distinction is relatively crude, I still believe that the 
distinction is a real, not an ostensible one, and that the two 'pure' paradigms 
are underpinned by quite different conceptions of the nature and status of 
knowledge. Before turning to a discussion of this issue, however, I should like 
to outline a model developed by van Lier (1988; 1990) for characterising 
applied linguistic research. 

Van Lier argues that applied linguistic research can be analysed in terms of 
two parameters: an interventfonist parameter and a selectivity parameter. 
Research is placed on the interventionist parameter according to the extent 
to which the researcher intervenes in the environment. A formal experiment 
which takes place under laboratory conditions would be placed at one end of 
thejnterventionist continuum/parameter, while a naturalistic study of a 
classroom in action would be placed at the other end of the continuum. The 
other parameter places research according to the degree to which the 
researcher prespecifies the phenomena to be investigated. Once again, a for- 
mal experiment, in which the researcher prespecifies the variables being 
focused on, would be placed at one end of the continuum, while an ethno- 
graphic 'portrait' of a classroom in action would occur at the other end of the 
continuum. Figure 1.3 illustrates the relationship between these two 
parameters. 

The intersection of these two parameters creates four 'semantic spaces': a 
'controlling' space, a 'measuring' space, an 'asking /doing' space, and a 
'watching 'space. The controlling space, which is characterised by a high 
degree of intervention and a high degreeof control, contains studies in which 
the experimenters focus their attention on a limited number of variables and 
attempt to control these in some way. For example, in an investigation into 
the effect of cultural knowledge on reading comprehension, the investigator 
may set up an experiment in which subjects from different cultural back- 
grounds read texts in which the content is derived from their own and other 
cultures. In such an experiment, the focus is on a single variable (cultural 
background) which is controlled through the reading texts administered to 
the subjects. 

The measuring space encloses those research methods involving a high 
degree of selection but a low degree of control. 'One selects certain features, 
operationally defines them, and quantifies their occurrence, in order to estab- 
lish a relationship between features, or between features and other things. 



PURE FORMS 

Paradigm 1 : exploratory-interpretive 

1 non-experimental design 

2 qualitative data 

3 interpretive analysis 

Paradigm 2: analytical-nomoiogical 

1 experimental or quasi-experimental design 

2 quantitative daia 

3 statistical analysis 

MIXED FORMS 

Paradigm 3: experimental-qualitative-interpretative 

1 experimental or quasi-experimental design 

2 qualitative data 

3 interpretive analysis 

Paradigm 4: experimental-qualitative-statistical 

1 experimental or quasi-experimental design 

2 qualitative data 

3 statistical analysis 

Paradigm 5: exploratory-qualitative-statistical 

1 non-exfjerimental design 

2 qualitative data 

3 statistical analysis 

Paradigm 6: exploratory-quantitative-statistical 

1 non-experimental design 

2 quantitative data 

3 statistical analysis 

Paradigm 7: exploratory-quantitative-interpretive 

1 non-experimental design 

2 quantitative data 

3 interpretive analysis 

Paradigm 8: experimental-quantitative-interpretive 

1 experimental or quasi-experimental design 

2 quantitative data 

3 interpretive analysis 

Figure 1.2 Types of research design (from Grotjahn 1987: 59-60) 
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MEASURING 



nonintervention 



WATCHING 



non- 
selective 



Figure 1.3 Parameters in research design (after van Lier IVSS) 



such as educational outcomes' (van Lier 1990: 34). For example, tiic 
researcher may be interested in the effect of teacher questions on student 
responses. Armed with a taxonomy of teacher questions, the rcsiMrclior 
observes a series of classes, documenting the types of questions askcil and tin- 
length and complexity of the responses. Here the researcher is highly selective 
in what he or she chooses to look at or for, but does not attempt to control 
the behaviour of either the teacher or the students. 

The asking/doing space contains studies in which there is a high degree of 
intervention, but a low degree of control. 'One investigates certain problem 
areas by probing, trying out minor changes, asking for participants' views and 
concerns, and so on. After a while it may be possible to pinpoint the problem 
so precisely that a controlled environment can be created in order to conduct 
an experiment, thus moving from [asking/doing] through watching to con- 
trolling. On the other hand, increased understanding through interpretation 
can also make experimentation unnecessary' (van Lier 1990: 34-35). 

The final semantic space, watching, is characterised by a lack of selectivity 
and a lack of intervention. The researcher observes and records what happens 
without attempting to interfere with the environment. Additionally, the 
researcher does not decide which variables are of interest or of potential sig- 
nificance before engaging in the research. While some form of quantification 
or measurement may be used, it is seen as no more than one tool among many, 
and not inherently superior to any other way of analysing data. An example 
of a study fitting into this final semantic space would he one in which the 
researcher wishes to provide a descriptive and interpretive portrait of a school 
community as its members go about their business of living and learning 
together. 

1 find van Lier's model of types of research a useful one, although, as van 
Lier himself points out, it is a simplification of what really happens when 
research is carried out. In reality, a particular piece of research may well tran- 
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sccnd its initial 'semantic space'. An investigation may well begin in the 
'watching 'space, and then, as issues emerge, the focus may become narrower. 
The researcher may then decide to establish a formal experiment to test an 
hypothesised relationship between two or more variables. In this instance, the 
research will have moved from the 'watching'space to the 'controlling' space. 
Regardless of the fact that it is a simplification, it does serve to highlight two 
of the most important questions researchers must confront at the beginning 
of their research, namely: 

- To what extent should I attempt to prespecify the phenomena under 

investigation? 

- To what extent should I attempt to isolate and control the phenomena 
under investigation? 

Brown (1988) provides a very different introduction to research from van 
Lier, being principally concerned with quantitative research. In his frame- 
work for analysing types of research, he draws a distinction between primary 
and secondary research. Secondary research consists of reviewing the litera- 
ture in a given area, and synthesising the research carried out by others. Nor- 
mally, this is a necessary prerequisite to primary research, which 'differs from 
secondary research in that it is derived from the primary sources of infor- 
mation (e.g., a group of students who are learning a language), rather than 
from secondary sources (e.g., books about students who are learning a lan- 
guage)' (1988: 1). Hence, it has the advantage of being closer to the primary 
.source of information. Primary research is subdivided into case studies and 
statistical studies. Case studies centre on a single individual or limited number 
of individuals, documenting some aspect of their language development, usu- 
ally over an extended period of time. Statistical studies, on the other hand, 
arc basically cross-sectional in nature, considering 'a group of people as a cross 
section of possible behaviors at a particular point or at several distinct points 
in time. In addition, statistical analyses are used in this approach to estimate 
the probability, or likelihood, that the results did not occur by chance alone' 
(p. ?)). In Brown's model, statistical studies arc further subdivided into survey 
studies and experimental studies. Survey studies investigate a group's atti- 
tudes, opinions, or characteristics, often through some form of questionnaire. 
Fxpcrimentai studies, on the other hand, control the conditions under which 
the behaviour under investigation is observed. 

f-'or iiist.incc, a rcsc.irchcr might wish to study the effects of being male or female on 
snuicnts' performance on a language pl.iccnicnt test. Such research might involve 
administering the test to the students, then separating their scores into two groups 
according to gender, and finally studying the similarities and differences in behavior 
Ixtwccn the two groups. Another type of experimental study might examine the 
rciatioiisiiip Ix'twceii students' scores on a language aptitude test and their actual 
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Types of research 



primary secondary 



statistical 



survey experimental ^ ' 

Figure 1,4 Types of research (after Brown 1988) 

performance in language classes, as measured by course grades. Experimental 
Mudics, then, can be varied in the types of questions being asked ... (p. 3) 

Brown's characterisation of types of research is set out in Figure 1.4. 

According to Brown, experimental research should exhibit several key 
characteristics. It should be systematic, logical, tangibJe, replicable, and 
reductive, and one should be cautious of any study not exhibiting these char- 
acteristics. A study is systematic if it follows clear procedural rules for the 
design of the study, for guarding against the various threats to the internal 
and external validity of the study, and for the selection and application of 
statistical procedures. A study should also exhibit logic a the step-by-step 
progression of the study. Tangible research is based on the collection of data 
from the real world. 'The types of data are numerous, bur chey are all similar 
in that'they must be quantifiable, that is, each datum must be a number that 
represents some well-defined quantity, rank, or category' (p. 4). Replicability 
refers to the ability of an independent researcher to reproduce the study under 
similar conditions and obtain the same results. In order for a reader to eval- 
uate the replicability of a study, it should be presented clearly and explicitly. 
Reductivity is explained in the following way: '. . . statistical research can 
reduce the confusion of facts that language and language teaching frequently 
present, sometimes on a daily basis. Through doing or reading such studies, 
you may discover new patterns in the facts. Or through these investigations 
and the eventual agreement among many researchers, general patterns and 
relationships may emerge that clarify the field as a whole' (p. 5). Most of these 
characteristics can ultimately be related to issues of validity and reliability, 
and we shall look in detail at these critical concepts later in the chapter. Table 
1. 1 summarises the key characteristics of good experimental research accord- 
ing to Brown. 

In this section I have reviewed the recent literature on research traditions in 
applied linguistics. My main point here is that, while most commentators 
reject the traditional distinction between qualitative and quantitative 
research as being simplistic and naive, particularly when it comes to the anal- 

Q 



r 

case 
study 



TABLE 1.1 CHARACTtRISTlCS OF GOOD KXPtRIMEN TAt. REStARCM 



Characteristic 



Key question 



Systematic Does the study follow clear procedural rules? 

Logical Does the study proceed in a clear step-by-step fashion, front 

question formation to data ct)Ilc'ction and analysis? 

Tangible Are data collected from the real world? 

Replicable Could an independent researcher reproduce the study? 

Reductive Does the research establish patterns and relationships among 

individual variables, facts, and observable phenomena? 



Source: Based on Brown (1988). 



ysis of published research, the distinction between the research traditions per- 
sists. Ultimately, most researchers will admit to subscribing to one tradition 
rather than another. How, then, are we to account for the persistence of a 
distinction which has been so widely criticised? 



The status of knowledge 

One reason for the persistence of the distinction between quantitative and 
qualitative research is that the two approaches represent different ways of 
thinking about and understanding the world around us. Underlying the 
development of different research traditions and methods is a debate on the 
nature of knowledge and the status of assertions about the world, and the 
debate itself is ultimately a philosophical one. It is commonly assumed that 
the function of research is to add to our knowledge of the world and to dem- 
onstrate the 'truth' of the commonsense notions we have about the world. 
(You might recall the statements made by students of research methods, some 
of which are reproduced at the beginning of this chapter.) In developing one's 
own philosophy on research, it is important to determine how the notion of 
'truth' relates to research. What is truth? (Even more basically, do we accept 
that there is such a thing as 'truth*?) What is evidence? Can we ever 'prove' 
anything? What evidence would compel us to accept the truth of an assertion 
or proposition? These are questions which need to be borne in mind con- 
stantly as one reads and evaluates research. 

In a recent television advertising campaign, the following claim was made 
about a popular brand of toothpaste: 'University tests prove that Brand X 
toothpaste removes 40% more plaque'. (The question of 40% more than 
what is not addressed.) By invoking the authority of 'university tests' the 
manufacturers are trying to invest their claim with a status it might otherwise 
lack. There is the implication that claims based on research carried out in 
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universities are somehow more 'scientific' and therefore beiiovahlc than 
claims made on the basis of anecdotes, the experience of the layperson, or the 
in house research of the manufacturers themselves. According to Winograd 
and Hlores (1986), the status of research based on 'scientific' expcrinu-iits and, 
indeed, the rationalist orientation which underlies it, is based on the success 
of modern science. 

The rationalist orientation ... is also regarded, perhaps k-causeof the prestige and 
success that modern science enjoys, as the very paradigm o( what it intaiis to think 
and be intelligent. ... It is scarcely surprising, then, that the rationalistic 
orientation pervades not only artificial intelligence and the rest of computer 
science, but also much of linguistics, management theory, and cognitive science . . . 
lationalistic styles of discourse and thinking have determined the questioits that 
have been asked and the theories, methodologic-s, and assumptions that h.we kvii 
adopted, (p. 16) 

I he following assertions have all been made publicly. You might like to con- 
sider these, and the evidence on which they are based, and reflect on which 
de^^rve to be taken seriously on the balance of the evidence provided. 

ASSERTION I 

Second language learners who identify with the target culture will master the 
language more quickly than those who do not. (Evidence: A case stiiiiy of an 
unsuccessful language learner.) 

ASSKRTION 1 

Schoolchildren are taught by their teachers they they need not obey their par- 
ents. (Evidence: A statement by a parent on a radio talk-back program.) 

ASSERTION 3 

Immigrants are more law abiding than native-born citizens, (llvideiice: An 
analysis of district court records.) 

ASSERTION 4 

IX-af children are more successful in school if their parents do not succumb 
to a sense of powerlessness when they experience difficulty communicating 
with their children. (Evidence: A study based on data from 40 deaf and 20 
hearing children.) 

ASSERTION 5 

.Affective relationships between teacher and students influence proficiency 
gains. (Evidence: A longitudinal ethnographic study of an inner city high 
school class.) 
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ASSERTION 6 

Students who nrc taught formal grammar develop greater proficiency than 
students who arc taught through 'immersion' programs. (Evidence: A formal 
experiment in which one group of students was taught through immersion 
and another group was taught formal grammar.) 

In actual fact, all of these assertions can be challenged on the basis of the evi- 
dence advanced to support them. Some critics would reject assertions 1, 2, 
and 5 on the grounds that they are based on a single instance (in the case of 
1 and 2 on the instance of a single individual, and in the case of 5 on the 
instance of a single classroom). Such critics would argue that the selection of 
a different mdividual or classroom might have yielded a very different, even 
contradictory, response. (We shall return to the issues of 'representativeness' 
and 'typicality' of data again in later chapters, particularly Chapter 3 on eth- 
nography, and (Chapter 4 on case study.) Assertion 3 could Ix; challenged on 
the grounds that the causal relationship between fewer court convictions and 
demographic data has not been demonstrated. (It might simply be, for exam- 
ple, that criminals from immigrant communities are smarter, and therefore 
less likely to Ix; caught than native-born criminals.) The problem with this 
study is that we can account for the outcomes through explanations other 
than the one offered by the researchers. Someone versed in research methods 
would say that the study has poor internal validity. (We shall look at the ques- 
tion of validity in the next section.) Assertion 4 might be criticised on the 
grounds that 'power' and 'powerlcssness' have not been adequately defined. 
Such a criticism is aimed at the construct validity of the study. (We shall also 
look at issues related to constructs and construct validity in the next section.) 
The final assertion can 1^ challenged on the grounds that the two groups 
might not have been equal to begin with. 

In the final analysis, the extent to which one is prepared to accept or reject 
particular methods of inquiry and the studies utilising these methods will 
depend on one's view of the world, and the nature of knowledge. For some 
[)eoplc the notion that there arc external truths 'out there' which are inde- 
pendent <)f the olwervcr is self-evident. For others, this notion, which underlies 
the qu.iinitative approach to research, is questionable (sec, for example, Win- 
ograd and Florcs 1986). 

Some key concepts in research 

In this section, we shall look in greater detail at some key concepts which 
have to this point only been touched on in passing. We shall look in particular 
at the concepts of reliability and validity. First, however, 1 should like briefly 
to discuss two other terms. These are deductivism and inductivism. 



An introduction to research methods and traditions 

Two procedures open to researchers are inductivism and deductivism. 
Deductive researchhe^ms with an hypothesis or theory and then searches for 
evidence either to support or refute that hypothesis or theory. Inductivism 
seeks to derive general principles, theories, or 'truths' from an investigation 
and documentation of single instances. Numerous commentators have criti- 
cised what is called naive inductivism (see Chalmers 1982), which is the belief 
that we can arrive at the 'truth' by documenting instances of the phenomenon 
under investigation. Popper (1968, 1972) illustrated the naivety of inductiv- 
ism with his celebrated swan example. He pointed out that we are never enti- 
tled to make the claim that 'All swans are white', regardless of the number of 
sightings of white swans. Though we may have sighted one thousand white 
swans, there is nothing to say that the one thousand and first sighting will 
not be a black swan. This led Popper to advance his falsificationist principle. 
This principle states that while we can never conclusively demonstrate truth 
through induction, we can in fact falsify an assertion through the documen- 
tation of a single disconfirming instance (as in the case of the black swan). 
According to Popper, all hypotheses should therefore be formulated in a way 
which enables them to be falsified through a single disconfirming instance. 
Taken to its logical conclusion, this view would have it that all knowledge is 
tentative and that, in fact, 'absolute truth' is an ideal which can never be 
attained. 

Chalmers (1982) introduces the falsificationist s position in the following 
manner: 

According to faisificationism, some theories can be shown to be false by an appeal to 
the results of observation and experiment. I have already indicated in Chapter 2 
that, even if we assume that true observational statements arc available to us in 
some way, it is never possible to arrive at universal laws and theories by logical 
deductions on that basis alone. On the other hand, it is possible to perform logical 
deductions starting from singular observation statements as premises, to arrive at 
the falsity of universal laws and theories by logical deduction. . . . The 
falsificationist sees science as a set of hyfxjtheses that are tentatively proposed with 
the aim of accurately describing or accounting for the behaviour of some aspect of 
the world or universe. However, not any hypothesis will do. There is one 
fundamental condition that any hypothesis or system of hypotheses must satisfy if 
it is to be granted the status of a scientific law or theory. Jf it is to form part of 
science, an hypothesis must be falsifiable. (pp. 38-39) 

The argument that progress in applied linguistics should be through the for- 
mulation and testing of hypotheses which are falsifiable has been advanced 
by numerous researchers. Pienemann and Johnston (1987) mount a vigorous 
attack on a major and influential research program in applied linguistics on 
the basis that it is not falsifiable. McLaughlin (1987) also argues that falsifi- 
ability or disconfirmation is the most important means to achieving scientific 
progress in applied linguistics. 
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III any scientific endeavour the number of potentially positive hypotheses very 
greatly exceeds the number of hypotheses that in the long run will prove to be 
c<)nip.itible with t)bservations. As hypotheses are rejected, the theory is either 
disconfirmed or escapxs from being disconfirmed. The results of observation 'probe' 
but do not 'prove' a theory. An adequate hypothesis is one that has repeatedly 
survived such probing - but it may always be displaced by a new probe. 
(McLaughlin 1987: 17) 

in reality, comparatively few hypotheses in applied linguistics can be demol- 
ished by a single disconfirming instance. In most cases we are interested in 
general trends and statistical tendencies rather than universal statements. 
Even researchers who claim their research is falsifiable have ways of protect- 
ing their theories from attack. For example, some second language acquisi- 
tion researchers (see, for example, i'ienemann and Johnston 1987) claim that 
the morphosyntax of all learners of English as a second language passes 
through certain developmental stages. These stages are defined in terms of the 
morphosyntactic items that learners are able to control at a particular stage, 
which in turn are governed by speech-processing constraints. According to 
the researchers, it is impossible for learners to 'skip' a stage, and if a single 
learner were to be found who had mastered, say, a stage 4 grammatical item 
while still at stage 2, then the developmental hypothesis would have been fal- 
sified. In fact, when such instances occur, it may be claimed that the learners 
in question have not really internalised the item but are using it as a formulaic 
utterance. Given the difficulty in determining with certainty whether or not 
an item is or is not a formulaic utterance, it is highly unlikely that the theory 
will ever be falsified. 

Two terms of central importance to research are reliability and validity, 
and I shall return ro these repeatedly in the course of this book. Reliability 
refers to the consistency of the results obtained from a piece of research. 
Validity, on the other hand, has to do with the extent to which a piece of 
research actually investigates what the researcher purports to investigate. It 
is customary to distinguish between internal and external reliability and 
validity, and I shall deal with each of these briefly in this section. The descrip- 
tion and analysis provided here is developed and extended in subsequent 
chapters. 

Reliability refers ro the consistency and replicability of research. Internal 
reliability refers to the consistency of data collection, analysis, and interpre- 
tation. External reliability refers to the extent to which independent research- 
ers can reproduce a study and obtain results similar to those obtained in the 
original study. In a recent investigation into classroom interaction, one of my 
graduate students coded the interactions of three teachers and their students 
using an observation schedule developed for that purpose. I also coded a sam- 
ple of the interactions independently. When the student and 1 compared the 
categories to which we had assigned interactions, we foimd that we were in 
agreement in 95% of the cases. We took this high level of agreement as an 
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indication that this aspect of the study had high internal leliahility. If a sec- 
ond graduate student were to conduct the study a sccoini time .uul oinaiii I lie 
same results, we could claim that the stutly was externally reliable. (1 his 
'inter-rater reliability' procedure is but one way of guarding .igainst threats 
to the internal reliability of a study. We shall consider alternative procedures 
in Chapter 3.) 

There are two types of validity: internal validity and external validity. 
Internal validity refers to the interpret.ibility of research. In ex()erimeiu.il 
research, it is concerned with the question: Can any ditfereiices which .ire 
found actually be ascribed to the treatments under scrutiny? I-.xtcriial validity 
refers to the extent to which the results can W generalised from samples to 
po^julations. Researchers must constantly be alive to the potential anil actual 
threats to the validity and reliability of their work. Table 1.2 provides twt) 
sample studies which illustrate the threats to validity posed by [joor research 
design. 

One of the problems confronting the researcher who wishes to gu.ird 
against threats to external and internal validity is that iiicasnres to strengthen 
internal validity may weaken external validity and vice versa, as Beretta has 
shown. 

Internal validity lias to do with factors which may directly aflect oiitcoims, while 
rxiernal validity is concerned with gcncralisability. If all variables, such as 
treatments and sampling of subjects, are controlled, then we might say thai 
laboratory conditions pertain and that the experiment is more likely lotv 
internally valid. However, what occurs under such conditions may not occur in 
typical circumstances, and the question arises as to liow far we may generalise Irom 
the results. (Beretta 1986a: 297) 

However, if the researcher carried out the snuiy in context, this may increase 
the external validity but weaken the internal validity. 

In addition to internal anti external validity, researchers need to jjay close 
attention to construct validity. A construct is a psychological quality, such as 
intelligence, proficiency, motivation, or aptitude, that we cannot directly 
observe but that we assume to exist in order to explain behaviour we cm 
observe (such as speaking ability, or the ability to solve problems). It is 
extremely imfxjrtant for researchers to define the constructs they are inves- 
tigating in a way which makes them accessible to the outside observer. In 
other words, they need to dcscrilie the characteristics of the constructs in a 
way which would enable an outsider to identify these characteristics if they 
came across them. If researchers fail to provide specific definitions, then we 
need to read between the lines. For example, if a study investigates 'listening 
comprehension', and the dependent variable is a written cloze test, then the 
default definition of 'listening comprehension' is 'the ability to complete a 
written cloze passage'. If we were to find such a definition unacceptable, we 
would be questioning the construct validity of the study, (.onstruct validity 
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For example, because the class was in progress for several weeks between steps 
4an(.l 5, itconkl he argued that improvements were simply due to maturation, 
ami that progress would have been recorded regardless of the tyjjes of ques- 
rioiis askeil by the teacher. Despite these problems, I still believe that the inves- 
tination was worth carrying out and reporting, particularly as it is the sort of 
investigation whicli can be replicated rather easily by other teachers. 



Conclusion 

In this chapter, I have dealt with .some of the central themes and issues asso- 
ciated with research into language learning and use. I have argued that, while 
the distinction between qualitative and quantitative research is simplistic in 
many ways, it does represent a real, not an ostensible, distinction. However, 
the (iisrinction is a philosophical one which is not always reflected in the 
actual conduct of empirical investigation. Underpinning quantitative 
research is the positivistic notion that the basic function of research is to 
inicover facts and truths which are independent of the researcher. Qualitative 
researchers question the notion of an objective reality. As Rist a.sserts: 

Uitiuiatrly, ilic issue is not research .strategics, per sc. Rather, the adherence to one 
para<iigin as op[X)srJ to another prcdi.spcjscs one to view the world and the events 
within it in profoundly different ways. (1977: 4.3) 

In the chapters which follow, we shall take up and explore these issues in 
greater detail. Chapter 2 provides an introduction to the use of the experi- 
mental method. We slvail also l(K)k at the use of statistics and the logic of 
inferential statistics, which efiables us to make generalizations beyond the 
subjects we have studied to a wider population. Issues assf)ciated with 
descriptive and iiiter[>ri'tive research are taken up in Chapter 3, which looks 
at ethnography, and Cihapter 4, which deals with case study methodology, 
incluiiing single case research. Chapter 5 looks at aims, issues, and methods 
in classroom observation. In Chapter 6, the focus of concern is introspection 
and the use of introspective methods in research, including think-aloud tech- 
niques, diaries, anil retros()ection. The focus of C'hapter 7 is the collection ami 
an.ilysis of speech data collected in naturally occurring interactions. The 
theme of Cihapter 8 is elicitation, and the chapter deals with a number of dif- 
ferent methoils, such as the interview and questionnaire, which arc designed 
ti) elicit data from language learners and users. Chapter 9 looks at some of 
the theoretical and practical issues involved in program evaluation, and raises 
the ijuestion of whether or not program evaluation is a form of research. In 
ihe final chapter, practical questions associated with the formulation of a 
research question or hypothesis, the selection of an appropriate research 
design, and the analysis and presentation of data arc taken up. 
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Questions and tasks 

1. Complete the following statements 
Research is . . . 

Research is carried out in order to . . . 

2. Here is a list of questions which have been addressed in the research lit- 
erature, (a) What are the key constructs associated with each question? (b) 
Which do you think might best be investigated through some form of exjjer- 
iment, and which might best be investigated through naturalistic investiga- 
tion? (c) Can you find any studies which might be investigated either through 
an experiment or a naturalistic study? 

- Are authentic materials more effective in bringing about learning than 
materials written specifically for the classroom? 

- Docs learning a second language involve the same psycholinguistic pro- 
cesses as learning a first language? 

- Are there significant differences in the ways in which people interact with 
members of the same/opposite sex? 

- Do learners from the same ethnic background share learning strategy 
preferences? 

- In classrooms containing both first (LI) and second (L2) language learners, 
should teachers use different language and interactional patterns with LI 

and L2 speakers? 

- Do learners who have grammatical rules explained to them learn more 
effectively than those who learn inductively? 

- What happens when teachers share decision-making with their learners? 

- Is there a positive correlation between the language addressed to a child in 
its preschool years by the primary caregiver and ultimate academic 
attainment? 

- Is the difficulty of a listening text influenced by the listener s background 
knowledge of the subject in question? 

- How do people keep casual conversations going? 

- Is there a 'critical period' for language acquisition, after which it is much 
more difficult to acquire a second language? 

Do children consciously try and work out rules as they acquire their first 
language, or is it a subconscious process? 

How are power relationships in the multilingual workplace linguistically 
marked? 

Are doctors who are trained in the language and culture of patients from 
different ethnic backgrounds able to diagnose more effectively? 
Do first language learners learn to do discourse before they learn grammar? 
How do parents help their children acquire language? 
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3. Which of the following statements from Chalmers (1982) are falsifiable 
and which are not? 

a. It never rains on Wednesdays. 

b. All substances expand when heated. 

c. All points on a Euclidean circle are equidistant from the centre. 

d. Heavy objects, such as a brick, when released near the surface of the Earth 
fall straight downwards if not impeded. 

e. When a ray of light is reflected from a plane mirror, the angle of incidence 
is equal to the angle of reflection. 

f. Luck is possible in sporting speculation. 

There are some types of applied linguistic research in which a single discon- 
firming instance is sufficient to invalidate the claim, hypothesis, or theory 
under investigation. Which of the following statements would you accept as 
being invalidated by the existence of a single disconfirming instance? 

- Learners will acquire the ability to form questions through inversion before 
they acquire Wh- questions formed through 'do' insertion. 

- Authentic listening materials are more effective than materials specially 
written for the classroom. 

- Parents of hearing and hearing-impaired children will code-switch to 
accommodate the hearing status of the child. 

- The degree to which a learner acculturates to the target language group 
will control the degree to which he or she acquires a second language. 

(In the first part of the preceding task, assertions c and f are not falsifiable. 
Assertion c is a definition and therefore a necessary truth. Assertion f is 
quoted from a newspaper horoscope, and, as Chalmers (1982: 40) says: 'It typ- 
ifies the fortune-teller's devious strategy. The assertion is unfalsifiable. It 
amounts to telling the reader that if he has a bet today he might win, which 
remains true whether he bets or not, and if he does, whether he wins or not.') 
4. Review one or more studies concerned with some aspects of language 
learning and use which has been published in a language journal such as Lan- 
guage Learning, Modern Language Journal, Applied Linguistics, TESOL 
Quarterly, Canadian Modern Language Review.) ALT Journal, or Studies in 
Second Language Acquisition. Make a note of the functions of the following 
components of the report - in other words, what is the author trying to do 
in each of these sections? (Not all reports will necessarily contain all of these 
elements, which is why you may need to look at several.) 

Abstract 

Introduction 

Rationale 

Literature review 

Hypothesis or research questions 
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Data collection instruments or methods 

Research procedure 

Subjects 

Data analysis 

Results 

Discussion 

Conclusions 

5. What do you see as the potential threats to the validity and reliability »)f 
the action research project described in Figure 1 .5? What steps might Ix' taken 
to guard against these threats? 

6. What are some of the questions, issues, or problems from yt)ur own pro- 
fessional context which might be investigated through action research? 

Further reading 

Chalmers (1982) provides a derailed intrcxluction to the nature and philoso- 
phy of scientific research. He deals at some length with the problems of deduc- 
tion, induction, and falsifiability. 

Nunan (1989) is intended as a practical introduction to action research lor 
those classroom practitioners interested in carrying out such research in their 
own classrooms. A useful collection of papers on action research can he found 
m Lomax (1989). 

Chaudron (1988) provides an extremely detailed and comprehensive intro- 
duction to issues in second language classroom research. Although the 
emphasis in the b(X)k tends to be towards quantitative rather than qualitative 
research, both are dealt with. 

For an introduction to the ethnography of classroom research, sec van Lier 
(1988). Key studies in classroom research are publi-shed with a critical com- 
mentary in Allwright (1988). 



23 



2 The experimental method 



Thomas GrnJgrind, sir. ... A man of facrs and calculations. A man who proceeds 
ufxjn the principle that two and two are four, and nothing over, and who is not to 
be tnliced into allowing for anything over. Thomas Gradgrind, sir - peremptorily 
I homas - 7"homas Gradgrind. With a rule and a pair of scales, and the 
multiplication table always in his pocket, sir, ready to weigh and measure any 
parcel of human nature, and tell you exactly what it comes to. 

(Charles Dickens, Hard Times) 

One popular iniapc of research is that it is concerned with forma! experiments 
of various types (recall the statements of students at the beginning of Chapter 
1). In this book wc shall see that although cxp)criments are impwrtant, they 
arc by no means the only way in which research can or should be conducted. 
In this chapter, wc shall consider what is meant by the experimental method. 
Wc siiall also consider the use of statistics in research, and look at some of the 
more commonly employed statistical tools in applied linguistics. I should 
make clear at the outset that this chapter is not intended to teach the reader 
how to 'do' statistics. Rather, it is intended as a guide to the basic concepts 
needed to read with sotiic understanding research reports utilizing statistics, 
and to appreciate the logic lx:hind the use of statistical inference. For a more 
detailed introduction to statistics in applied linguistic research, see the ref- 
erences cited in the 'Further Reading' section at the end of this chapter. 
This chapter addresses the following questions: 

- What are variables, samples, and populations, and why are they important 

in research? 

- What arc the basic principles of sound experimental design.' 

- What do we mean by inferential statistics? 

- When is it appropriate to use the following statistical procedures: t-test 
analysis of variance, correlation, chi-square? 

- What is the difference lx:twccn true experiments, quasi-experiments, and 
pre-experimentsf 

The context of experimentation 

What arc the contexts in which an experiment is the appropriate method for 
collecting and analysing data? Generally speaking, exp)crimcnts are carried 
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TABLE 2.1 TYPES OF VARIABLES USED IN LANGUAGE RESEARCH 



Ty-pe 


Example 


Nominal 


LI background: e.g., Arabic, Spanish, etc. 


Ordinal 


Rank on a test of grammar: e.g., first, second, third, etc. 


Interval 


Numerical score on standardised language test 



out in order to explore the strength of relationships between variables. A vari- 
able, as the term itself suggests, is anything which does not remain constant. 
In our case, it includes language proficiency, aptitude, motivation, and so on. 
Language researchers often want to look at the relationship between a vari- 
able such as a teaching method and a second variable, such as test scores on a 
formal test of language proficiency. In such a case it is customary to distin- 
guish between the two variables by giving them different labels. The label 
given to the variable that the exf)erimenter expects to influence the other is 
called the independent variable. In our case this would he the teaching 
method. The variable upon which the independent variable is acting is called 
the dependent variable - in our case, the test scores. 

Variables can also be classified according to the type of scale on which they 
are measured (see Table 2.1). A nominal scale measures mutually exclusive 
characteristics, such as sex and eye colour. (A subject cannot simultaneously 
belong to the category 'male' and the category 'female', or the category 'blue- 
eyed' and the category 'brown-eyed'.) Ordinal scales are for those variables 
which can be given a ranking, such as first, second, third, but in which the 
actual score itself is not given. An interval scale not only provides information 
on the rankings of scores, as does an ordinal scale, but also indicates the dis- 
tance between the scores. Most test score data are of this type. A final type of 
scale for measuring variables is the ratio scale, which measures absolute val- 
ues, such as temperature. Ratio scales are of little interest in applied linguis- 
tics, because variables such as language proficiency do not exist as absolute 
quantities; therefore, ratio scales will not be dealt with further here. 

Let us consider an example of a situation in which an experiment might 
be an appropriate way of gathering data. Imagine that you have developed 
some innovative listening materials for low level learners. You have used 
these materials with a range of classes, and believe that they are significantly 
sup)erior to the traditional materials which are used in your school. However, 
your colleagues are sceptical. How can you convince them that your mate- 
rials are more effective than the traditional ones? There are many ways in 
which you could collect evidence. You could survey the students through 
interviews and questionnaires, and obtain their subjective impressions. You 
could ask a sympathetic colleague to become a participant observer in your 
classroom and make an ethnographic record of the teaching and learning 
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going on. These measures, however, are unlikely to sway your sceptical col- 
leagues, who will be convinced only by test score data obtained through stan- 
dardized tests. 

You might be tempted to test your students at the end of the semester and 
present the results (assuming they are favourable) to your colleagues. How- 
ever, you come across the following attack on such an approach (which is 
rather contemptuously dismissed as 'one-shot research'): 

Much research in education today conforms to a design in which a single group is 
studied only once, subsequent to some agent or treatment presumed to cause 
change. Such studies might be diagramed as follows: 

XO 

[X = the treatment administered to the subjects, and O = the observation.] 
[ Unfortunately] . . . such studies have such a total absence of control as to be of almost 
no scientific value. ... It seems well-nigh unethical ... to allow, as theses or disser- 
tations in education, case studies of this nature (i.e., involving a single group observed 
at one time only). (Campbell and Stanley 1963: 176-177) 

Convinced by this argument, your next inclination is to test two groups, one 
which has used the innovative iT\aterials and one which has not. However, 
you quickly realize that it is no good simply testing the students at the end 
of the semester and comparing their scores with those obtained from another 
class at the same year level, because the groups might not have been at the 
same level to begin with. Fine, you might think, we can test both groups at 
the beginning of the term as well as the end. Then, if the group which has had 
the benefit of the innovative materials does better than the group that has used 
the traditional materials, we can presumably ascribe the superior perfor- 
mance to the materials. 

While your research design is becoming more rigorous, it is still not rig- 
orous enough to allow you to claim that there is a causal relationship between 
the independent variable (your innovative materials) and the dependent vari- 
able (the students' test scores). There is always the possibility that some factor 
other than the experimental materials has brought about the observed differ- 
ences in the scores. For example, you may have happened to select a group of 
fast track or high aptitude students as the recipient of the experimental mate- 
rials, and a group of slow learners as the 'traditional' group. In order to guard 
against such 'contamination', sound experimental design suggests that you 
should randomly assign students to either the control group, which uses the 
traditional materials, or the experimental group, which uses the innovative 
materials. You are then in a better position to argue that any differences on 
the end-of-term test are due to the experimental treatment (i.e., the innova- 
tive materials), because you can assume that other variables which might have 
an effect (such as intelligence or aptitude) exist in equal quantities in both the 
control and experimental groups, and therefore cancel one another out. You 
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should also test b<Mh groups o( students k-forc the expcrimenl jiiM to ni.ikc 
sure that the groups really are the same. 

If you carry out the procedures alreaily described, th.u is, r;iiulomly assign- 
ing your subjects to either the control or experinient.il group, and adnimiv 
lering a pre- and post-treatment test, then you could reasonably claim to iiave 
carried out what is known as a 'true' experiment. If you have not carried out 
these prtK-edures, then the internal validity of your experiment is under threat 
(recall the discussion on reliability and validity in Chapter I), k'c.uise some 
variable you have not controlled may be affecting the dependent variable. 
Recalling van Lier's model (see Figure l.,l), you can see from tliis description 
why the formal experiment belongs to the highly controlled/highly selective 
quadrant of the diagram. 

Unfortunately, it is not always practicable to rearrange students into dif- 
ferent groups or classes at will. There are rimes when, it we arc to carry (Uit 
an experiment at all, it will have to lie with intact groups ot subjects, that is, 
subjects who have been grouped together for reasons other than the carrynig 
out of an experiment. In these situations, while the internal validity of the 
experiment is weakened, it may still be thought desirable to proceed with the 
study. In instances such as this, researchers speak of quasi- or pre-experMiieiits 
rather than true experiments. We shall look a littler further at these diftcrent 
types of experiments later in this chapter. 

For argument's sake, let us imagine that you have been able to randomly 
assign sixty final-year secondary school students to control and experimental 
groups, and that a pre-test shows the two groups to Iv at the same level of 
proficiency. You teach both groups for a term, using the innovative materials 
with the experimental group and the traditional materials with the control 
group. At the end of the term, the groups are rete.sted, and you obtain the 
scores for each student. You work out the tnean, or average, for each group 
and obtain the following: 

C'ontrol group: 58 
Hxperimental group: 62 

The experimental group has, on average, outscored the control group. Are 
you therefore in a position to claim that the innovative ntateriais are superior 
to the traditional materials? Not yet. You have selected a sample, or subset, 
of all the possible students in the final year of secondary school who are study- 
ing your subject. If you tested them again tomorrow, or if you selected a sec- 
ond group of subjects and tested them, you would get different scores. There- 
fore, you need to use 'statistical inference' to work out whether the scores 
you obtained resulted from students' really being different, as suggested by 
the test scores, or whether the difference came about by chance or sampling 
variation. If all the students do share common, observable characteristics 
which differentiate them from other students, we say they represent a differ- 
ent population. A subset of individuals from a given population is a sumplc. 
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In order to illustrate the logic of inferential statistics, wc need to go back 
.1 step or two nnd consider a number of basic concepts. This we shall do in the 
next section. 

The logic of statistical inference 

The aim of riiis section is to introduce you to the logic of statistical inference. 
While the information in the section will not necessarily provide you with 
the skills needed to carry out statistically based research, it should help you 
to understand the logic Ixhind experimental research in which the researcher 
makes claims about an entire population based on data ol^taincd from a subset 
or sample of that population. 

In most research, it is not possible tocollect data from the entire population 
of individuals in which one is interested. Consider an investigation of the lis- 
tening proHcicncy of first-year secondary school French students. It would be 
extremely time consuming, although not im[K)ssiblc, to obtain data on all 
such students. Normally, someone wishing to carry out such an investigation 
would select a sample (say 30, 50, 100, 200) from the population and test 
these. However, a problem immediately arises: To what extent are the sample 
data ri-preseiuative of the population as a whole? Fortunately, certain pro- 
cedures exi.st which enable us to determine the probability that the sample 
does represent the population from which it is drawn. In order to appreciate 
tlK' logic behind these prcxedures, one must be familiar with the following 
statistical concepts; tueati, sItinJani dei'iation, normal distribution, and stan- 
diird error. 

From .1 statistical point of view, when studying numerical data of various 
sorts, the two things we will be most interested in arc the extent to which the 
data are .similar and the degree to which the data differ. The most frequently 
emi'loytd measure of similarity is the //itw;; (symbolised by X) which is simply 
the .ivcrage of a set of scores (obtained by adding the individual scores 
togcduT and tlividing by the total numlx;r of scores). It gives us information 
alx)iit the central tendeticy of the .scores. The standard deviation (SD), on the 
oilier hand, is the most important measure of dispersion, giving us informa- 
tion on the extent to which a set of scores varies in relation to the mean. It is 
calculated by deducting the mean from each individual score, squaring the 
resulting ligures to get rid of the minus signs, adding these together, and 
dividing by the mmibcr of scores minus one. (Dividing by one less than the 
number of scores is a correction for the fact that the variability of scores for 
a single group of subjects tends to be less than the variability for all possible 
scores.) I bis gives us the iniriaticc. By obtaining the square roox of this figure 
we arrive at the standard deviation. A simple example is set out in Table 2.2 
to demon.stratc the procedures involved. 

From Table 2.2, we sec that the variance for these scores is ,^.1 1 1, and the 
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TABLE i.Z CALCULATING THE STANDARD 
DEVIATION: A WORKED EXAMPLE 



Squares of figures 
Score Score — mean' in 2nd column 



2-2 4 

8 4 16 

5 1 1 

3 -1 1 

4 0 0 
1 -3 9 
2-2 4 
3 -1 1 

5 1 1 
7 3 J 

40 46 



The sum of the squares/(no. of scores minus 1): 
46/(10 - 1) = 5.111 

Variance = 5.111 

Standard deviation (i.e., square root of variance) = 2.26 

"The mean = 4. This is calculated by dividing the sum 
of the scores (40) by the total number of individual scores 
(10). 

square root of the variance is 2.26. Thus, the standardised amount by which 
these scores deviate from the mean is 2.26. 

As we shall see, in order to determine the probability that scores from two 
different samples come from two different fwpulations, it is crucial that we 
have information on the central tendency (in the form of mean scores) and 
dispersion of the scores (in the form of standard deviations). While it is clear 
that two sets of scores with widely differing means will have come from sam- 
ples from different populations, it is also the case that scores with widely dif- 
fering standard deviations would also indicate that they are from different 
populations. We need both types of information because scores can have very 
similar means, but quite different standard deviations, which would indicate 
that they come from different populations. 

Let us imagine that the following scores were obtained from some of the 
students in our experimental listening group: 
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There are several things to note about the scores. Firstly, and most obviously, 
the scores vary. Less obvious is the fact that the variation is systoniatic. In lact,' 
if we plot them as a frequency distribution, they are close to what is known 
as a normal distribution. This is demonstrated in Figure 2.1. Not all sets of 
data are normally distributed. However, if we have a set of data that are nor- 
mally distributed, then we can use the special characteristics of the distribu- 
tion to do a number of interesting things, including determining the proba- 
bility that our sample is representative of the population from which it was 
drawn. For our purposes, one of the most important facts about the normal 
distribution is that a predictable number of scores from the sample will be 
within limits established by the mean and standard deviation (see Figure 2 ■>) 
We can see from the figure that in any set of normally distributed scores, 68% 
of those scores will be within 1 SD of the mean, 95% of the scores will be 
within 2 SDs of the mean, and over 99% of the scores will be within 3 SI>j of 
the mean. 

Let us return to our listening example. I have calculated the mean and SD 
of scores from the fifty subjects making up the experimental group to be 6^ 
and 3.8, respectively. Plotting these on the figure showing the normal distri- 
bution provides us with Figure 2.3. We can use this information to make 
numerous observations. For example, we know that 95% of all scores will 
h\\ within the range 54.4-69.6, and that 97.5% of scores will fall below 69 6 
Therefore, a student scoring 70 or better will be in the top 2.5% of the group 
Armed with information about means and standard deviations we can 
analyse and compare numerical data in ways which are not possible with raw 
scores. (Raw scores are the actual scores obtained on tests, etc.) We can for 
instance, compare scores which have been derived from different sources 
Consider the case of a student who obtained a raw score of 90 in Japanese and 
80 in Chinese. We might be tempted to conclude that the student is better at 
Japanese than Chinese. However, raw scores do not allow us to come to such 
conclusions. If we know the mean and SD of the sample, we can draw con- 
clusions In this student's case, the mean and SD on the test of Japanese were 
60 and 15, and on the Chinese test, 65 and 5. Knowing this, we can say that 
the student did comparatively better in Chinese than Japanese. Why' Well 
TQn"°V characteristics of the normal distribution (ND) that a score 

of 90 m Japanese is 2 SDs above the mean, putting the student in the top 2 5% 
of the sample. However, the score of 80 in Chinese is 3 SDs above the mean 
putting the student in the top 1 % for that subject. 

In doing research in which the data are obtained from formal experiments 
It IS usua ly only feasible to test a sample of the population, for reasons out- 
lined earlier. The problem for the researcher is to decide whether the data 
obtained from the sample are representative of the population as a whole 
Imagme you are presented with the score of a single subject from our listening 
experiment. This student obtained a score of 58 on the listening test Given 



30 



Ihc cxfci iniciiidl /ncll.'ikl 











X 














X 


X 


X 










X 


X 


X 


X 








X 


X 


X 


X 


X 


X 




X 


X 


X 


X 


X 


X 


X 


X 


58 


59 


60 


61 


62 


63 


64 


65 



l igure 2. 1 Distribution of scores from hypotbelidil listening test 




figure 2.2 Percentage of scores falling ivitbin I, 2, and i standard 
deviations of tbe mean 
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50.6 
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+ 1 SD +2 SD +3 SD 
65.8 69.6 73.6 



Figure 2.3 Scores falling within I, 2, and i standard deviations of the 
mean: an example 



what we already know about the percentage of students falling witiiin I, 2, 
and 3 SDs of the mean, which of the following statistics would he more likely 
to represent the entire population from which this single subject lias been 
drawn? 
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B 
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50 


60 


70 


SD 
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The A statistics would imply that the student is more than 3 SDs above the 
mean, which wonld place him or her in the top .15%. The C statistics would 
place the student more than 2 SDs below the mean, and therefore in the lowest 
2.57o. The B statistics would place the student within 1 SD below the mean, 
and therefore much closer to the hypothesised mean than either the A or C 
statistics. It is therefore most likely that the B statistics represent the popu- 
lation as .1 whole for the simple reason that the further a score is from the 
overall population mean, the less likely it is to occur. 

The problem, of course, is in determining the critical statistics of mean and 
SD for the population. While we can never be certain of these data without 
testing the entire population (which we have already seen is generally not 
feasible), we can estimate them from sample data. Imagine we obtain the fol- 
lowing scores from several more students: 61, 60, 64, 57. We can now revise 
our estimation of the population mean, which, if reflected by our sample of 
5, would be 60, and the standard deviation, which would be around 2.75. The 
additional information enables us to modify our predictions about the pop- 
ulation as a whole. We can sec that the B statistics are now confirmed as most 
likely to represent the population from which the sample was drawn. This 
example demonstrates two points. Firstly, the more information we have, the 
more confident we can be about the accuracy of our predictions, and sec- 
ondly, wc can make predictions based on relatively small amounts of 
inft)rmati()n. 

If wc selected another five students from the population and tested them, 
wc would probably obtain different scores, with a different mean and stan- 
dard deviation - say, 61.5 and 3.1, respectively (possibly, we would even 
obtain different scores from the same students if wc tested them on another 
occasion). If wc kept selecting samples, calculating the means, and plotting 
these as ;i frequency distribution, wc would find that they arc normally dis- 
tributed, that their standard deviation is smaller than that of the individual 
samples, and that their mean is equal to the population mean. (It stands to 
reason that the means will have less variability than the individual scores.) 
Given the fact that the sample means arc normally distributed, we know that 
687(1 of all sample means will be within one standard deviation of the mean, 
that 957o of scores will be within 2 SDs of the mean, and that over 99% of 
sample nicans will be within 3 SDs of the mean. 

The standard deviation of sample means is known as the standard error. 
We can use this information to place a single sample mean in relation to the 
population mean from which it is drawn. For simplicity's sake, suppose that 
the mean of the .sampling distribution in our current example is 61 and the 
standard error is 1. Wc know that 68% of all sample means will be within 
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the range 60-62, that 957o of sample means svill l^e within the range 59-63, 
and that 99% of the sample means will be within the range 58-64. This means 
that if we drew 100 samples from the population, the mean for only one of 
these is likely to lie outside the range 58-64 (see Figure 2.4). 

The problem of course is that, as it is not a practical proposition to extract 
many samples from our population to test, we do not know the mean and 
standard error of the sampling distribution. However, we can estimate the 
standard error from a single sample by dividing the standard deviation of the 
sample by the square root of the number of observations in the sample (N); 
that is, the standard error equals the standard deviation divided by the square 
r(X3t of N. 

From the formula presented in the preceding paragraph, we can sec that 
the standard error will be affected by the size of the sample and the standard 
deviation of the sample. If we increase our sample size, the standard error will 
be reduced. If the standard deviation of the sample were larger than 3.8, then 
the standard error would also be larger. 

Thus far, we have studied a procedure for determining the extent to which 
we can say that a single sample mean is representative of the mean of the 
population from which it has been drawn. However, in most experimental 
research, we are interested not in a single mean, but in two or more means. 
Let us return to our listening example. With 100 subjects at our disposal, we 
have run a classical experiment and have done all the right things. For exam- 
ple, we have ensured that our subjects are randomly assigned to experimental 
and control groups, and wc have excluded a group of subjects who are judged 
to be fast learners. At the end of the experimental period we test both groups 
and obtain the data in Table 2.3. 

Although there is a difference between the mean scores of the two groups, 
we are not yet entitled to say that the difference indicates that the samples 
have been drawn from two different populations. As wc have already seen, 
variation between samples is to be expected. The question we need to settle 
statistically is the balance of probability that the two samples have been drawn 
from two different populations or from the same population. Another way 
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TABl.K 2.3 MEANS AND STANIMKO 
in VIATIONS FOR tXI'EKIMHNTAL AND 
CONTROL GROUPS 





Experimentdl 


Control 


Mean 


■ 62.0 


■ 58.0 


SI) 


3.8 


4.2 



of putting this is to ask whether, by exposing students ro the innovative mate- 
rials, we have created' another population, a population which is defined as 
'those students of a foreign language who have been exposed ro authentic 
listening materials'. 

We settle the issue statistically in the following manner. First we calculate 
the standard error for both groups by dividing the SD for each group by the 
square root of the number of subjects. Since there are 50 students in each 
group, and the square root of 50 is about 7, then we divide 3.8 and 4.2 by 7 
to obtain the standard errors (SE) for the experimental and control groups. 
This gives us the following: 

Experimental Control 
SE .5428 .6 

Knowing these figures, we work out how close our sample means are likely 
to be to the population mean from which they were drawn. We know that 
we can be 95% confident that the true population mean will be within two 
standard errors of the sample means. In the case of the experimental group 
this will be 62 plus or minus (.5428 X 2), which yields a range of 60.9 to 
63.085. For the control group, this is 58 plus or minus (.6 X 2), which yields 
a range of 56.8 to 59.2. This is represented in tabular form in Table 2.4. Figure 
2.5 provides a visual representation of these estimates. 

From these data, it looks as though, at the 95% level of confidence, we can 
be reasonably confident that the samples have been drawn from different pop- 
ulations. If must be kept in mind, however, that there is still a 5% chance in 
the case of both the experimental and the control groups that the true pop- 
ulation mean will lie outside the range we have established. 

I have worked through the examples in this section in some detail to give 
you an idea of the logic behind inferential statistics ('inferential' because we 
are 'inferring' from samples ro populations). Doing statistical calculations by 
hand is tedious, time consuming, and likely to result in error. Fortunately, 
computer software is now available for taking the pain out of such work. 
However, when doing such calculations, or when reading published studies 
employing statistics, it is important ro understand the logic behind them. 
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When comjjaring two means, as we have done in this sociion, the appro- 
priate test is a /-test, which carries out the .sorts of analytical promlun s we 
have just examined. The great disadvantage of the f-test is that it can a )m|j.ire 
only two groups. When comparing more than two means, or more than two 
groups, the appropriate test is the f"-tesr, which is based 011 a proccilure called 
analysis of variance (ANOVA). In order to illustrate the procedure, consider 
the following study. In this study, we are interested in the reading c()m()re- 
hension of first and .second language learners when they are dealing with aca- 
demic texts. In particular, we are investigating the cfteci <it memory coii- 
strainrs on the ability of first aiul second language learners to iiucgrare 
mformation within and Ix-yond sentences in a text. There are three groups of 
subjects: Group A = native speakers; Croup II - second lann\ia};c' learm rs; 
and Group C - foreign language learners. Each group is adniinisu rcd .1 li st 
consisting of a reading passage followed by two sets of conipi cliension ^iiies- 
tions. One set of questions requires the subjects to comprehend information 
within individual sentences in the test passage (we shall call this 'S-level' infor- 
mation), while the other set of questions requires the subjects to synthesise 
information located in different parts of the text (we shall call this ' I -level' 
information). Means and srandard deviations for the three groups are calcu- 
lated. The mean scores are re|jresented diagraminatically in Figure 2.6. 

From the data in Figure 2.6 it can be seen that, as might Iv expected, the 
first language group scored higher than the second language and foreign lan- 
guage groups on both sentence level and text level questions, and that tlie 
second language group scored higher than the foreign language group. W'e 
can also see that all groups did better on the S-level qnesrions than the 'I -levi-l 
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Figure 2.6 Comparison of mean scores for S- and T-lcrcl infonnation for 

three language groups 



questions. Further, the second and foreign language groups also seem to have 
(lone comparatively worse on the T-level questions than on the S-level ques- 
tions. 1 iowcvcr, are these differences significant.' An analysis of variance can 
help us answer this question. 

We have already seen that any set of scores from a given group of subjects 
will vary, as will the score from one group to another. Basically, the analysis 
of variance compares the variability of scores within groups with that 
between groups. First of all, the different language groups are compared, then 
the different question types are compared, and then the interaction between 
the groups and the question types arc compared. The computations result in 
an /-"-ratio, which we can use to consult a set of tables to determine whether 
the differences are significant. Table 2.5 shows what an ANOVA summary 
tabic looks like. 

The .inalysis of variance presented in Table 2.5 reveals that the difference 
l>etween the means of the language groups is significant at the 0.05 level of 
conh'dence (that is, we can Ix- ')5"/n conrident that the difference is significant), 
it can a Iso .seen that the difference between mca ns of the two different ques- 
tion types is significant. Finally, it shows that the interaction between subjects 
and question type is significant. This ability of ANOVA to test the signifi- 
cance of interactions between different variables is its great strength. In our 
example, it would .seem to indicate that second and foreign language learners 
have comparatively more difficulty than native speakers in integrating infor- 
mation from different sources within a text. Pedagogically, this would sug- 
gest that teachers should develop strategies which help non-native speakers 
comprehend beyond the level of the sentence. (Note that in the last sentence 
or two, 1 have gone beyond the evidence provided by the statistical data to an 
interpretation of the results.) 
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TABLE 2-5 ANALYSIS OF VARIANCE BETWEEN LANGUAGE GROUPS FOR S-LEVEL 
AND T-LEVEL QUESTIONS 



Source 


55 


df 


MS 


F 


Sig. 


Beltiven subjects 
Language groups (A) 
Error between 


39557.44 
137984.03 


1 
147 


19778.7 
938.67 


21.07 


0.05 


Within subjects 
Question types (B) 


22654.83 


1 


22654.83 


380.71 


0.05 


Interaction (AB) 


1069.20 


2 


534.60 


8.98 


0.05 


Error 2 


8747.47 


147 


59.51 







Key: 

SS = sum of squares 
df = degrees of freedom 
MS = mean squares 
F = the f-ratio 

Sig. = the significance level of the f-ratio 

Note: For further explanation of the terms used in this table, see Hatch and Lazaraton 
(1991). 

Additional statistical tools 

In the preceding section, the logic of statistical inference was illustrated with 
statistical tools for comparing means. While comparing means is one of the 
most common tasks confronting the language researcher, it is by no means 
the only task. Language researchers also want to do such things as compare 
the frequency with which certain events occur, and determine the extent to 
which one event is correlated with another. In this section we shall look at 
some of the tools that enable researchers to carry out these tasks. 

Experiments are very often concerned with the frequency with which 
events occur, rather than with the comparison of mean scores. For example, 
a researcher might be interested in testing the hypothesis that 'on task' behav- 
iour by second language learpers is higher in withdrawal classes ('withdrawal' 
or 'pullout' classes are those in which the second language learners are seg- 
regated from the native spe ikers) than in mixed classrooms, which contain 
both first and second language learners. In testing this hypothesis, an inde- 
pendent observer might observe a number of integrated and withdrawal class- 
rooms (either live or on video), noting each time that predefined on-task and 
off-task behaviours occur. (Students are 'on-task' when they are actively 
engaged in the pedagogic work of the day. They are 'off-task' when doing 
other things such as talking to friends, walking about the room, etc.) 
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TABI.K 2.6 FRKQUHNCY OF OFF-TASK AND ON-TASK 
BKHAVIOURS IN IN TKGRATFa) AND WITHORAWAl, 
t.l.ASSKS 





Integriili'ti 


Withdrawjl 


ToLll 


Off-task behaviours 


140 


60 


200 


On-task behaviours 


100 


100 


200 


Total 


140 


160 


400 



Note: This is called a 2 X 2 table kcause it shows the 
relationship between two variables with two levels. 



Such observations might result in a 2 X 2 table of the type set out in 
Table 2.6. 

We can deduce very little from such figures as they stand, apart from a few 
rather obvious facts (for example, that there is more on-task behaviour in the 
withdrawal classroom). We know from the discussion in the preceding sec- 
tion that it is highly unlikely that the observations will be identical. What we 
want to know is whether the difference is significant. The appropriate statis- 
tical tool for determining the significance or otherwise of such frequency 
counts is the chi-square test. I shall not go through the logic behind this test 
in the same detail as for the test of significance of the difference between two 
means, because it is similar to that behind the /-test and ANOVA. What fol- 
lows is an explanation of the procedure. 

If there is no association between the variables in our 2 X 2 tables (i.e., 
l^tween type of class and type of behaviour), we would expect the same pro-' 
portions to hold in each instance. Chi-square calculates the difference 
between the expected frequencies, if the proportions were the same, and com- 
pares these with the actual frequencies. It is then possible to determine 
whether or not these differences occurred by chance or whether there is a 
probability that the differences are significant. If there is no association 
between the variables, we would expect to obtain the following data. (These 
data are obtained by multiplying the individual totals and dividing by the 
grand total.) 

Table 2.7 shows us the frequencies we should expect if there is no associ- 
ation between type of class and task behaviour. The next step is to compare 
the expected values with the observed values, shown in Table 2.8. The bigger 
the discrepancy between expected and observed frequencies, the less likely it 
IS that they have been derived from the same population. (A fully worked 
example of chi-square can be found in Nunan 1989.) 

Another important family of statistical tests in applied linguistics belongs 
to the area known as correlation. These tests estimate the degree of associa- 
tion between two variables. For example we may wish to investigate the rela- 
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lABIF 2.7 FXI'KCTFin RHQIIKNCIFS 





liitf grated 


Withilriiit'iil 


•/•(./.»/ 


Ofl-Msk behaviours 


120 


«0 


2(H) 


(In-ijsk behaviours 


120 


80 


2(K) 


Total 


240 


160 


4(K) 


TABI.H 2.8 KXPKCTKl) (k) VKRSllS OBSi:RVtl) (O) 




tRFyiJKNCIKS 










Integrated 


WithdraWiil 


Total 


Oft-iask behaviours 


120(E) 


80 (F.) 


2(K) 




140 (O) 


60 (O) 




On-task behaviours 


120 (F.) 


80(K) 


KM 




100(C)) 


100 (O) 




Total 


240 


160 


4(X) 



tionship between speaking ability and reading ability in a foreign language. 
Or we might want to determine whether completing a cloze test raps the same 
underlying ability as completing a dictation test. We would carry our such 
studies by administering to a group of subjects tests which were assumed to 
tap the skills we were investigating, and then apply a statistical procedure to 
determine the extent to which a high score on one rest tended to go with n 
high score on the other test, a mid-level .score on one test went with a mid- 
level score on the other test, and so on. If the person obtaining the top .score 
on one test also obtained the top score on the second test and the person scor- 
ing second on one test scored second on the second test, and so on, all the way 
through the subjects, then we would iiave a case of perfect positive correlar 
lion. If the reverse were the case (rhs: person obtaining the top score on one 
rest obtained the lowest score on the .second test, and so on) we would have a 
f)erfect negative correlation. Of course it is extremely rare for researchers ro 
obtain perfect correlations. The extent of correlation is measured by a figure 
between — 1 and 1. Perfect positive correlation would result in a score of 1. 
Perfect negative correlation would result in a score of — 1. The procedure you 
are most likely to encounter in published research is the 'Pearson product- 
moment correlation'. 

In this section, and the one which preceded it, we have looked briefly at 
the most commonly employed types of statistical analysis: the /-test and 
ANOVA, which are u.sed to compar- mean.s; chi-square, for computing fre- 
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qiiencies; and correlation, which enables us to examine the degree of associ- 
ation l)ctween scores. In the next section, we look at types of experiments. 

Types of experiments 

Earlier in the chapter, I pointed out that it was not always feasible to carry 
out a true experiment - that circumstances such as the impossibility of ran- 
domly assigning subjects to experimental and control groups sometimes dic- 
tated that a quasi- or pre-cxperiment rather than a true experiment be con- 
ducted. (Of course, it is also possible that a quasi- or pre-expcriment is carried 
out bcciuisc of ignorance on matters of research design on the part of the 
researchers.) In this section, I shall describe the differences between these three 
types of experiments. The different designs are illustrated by Cohen and Man- 
ion (1985), who describe and critique three experiments reported in the lit- 
erature: the New Zealand Book Flood Experiment, the Bradford Book Flood 
Experiment, and the Understanding of Electrical Circuits Experiment. (I shall 
only dcscrik- the first of these studies. For an account and critique of the other 
two studies, sec Cohen and Manion 1985.) 

The New Zealand Book Flood Experiment was established to examine the 
effects of extensive reading and the wide availability of books on the reading 
habits and skills of primary school children. The two schools which took part 
in the study contained large numbers of Maori (New Zealand aboriginal) chil- 
dren who were believed to have limited access to books. The children were 
given sets of pre-expcrimental attitudinal, vocabulary and reading tests. They 
were then 'flooded' with books selected by teachers and librarians. Data 
includcil scores on posttests administered six months after the initiation of 
the project, as well as qualitative data on the reading habits of five children 
from each gri)up. Informal reading inventories were also used to obtain infor- 
mation on the reading lx;haviours of the subjects. Dependent variables 
included the quantity of reading undertaken and the skills, interests, and atti- 
tudes of t he subjects. Subjects showed significant improvements on all of these 
measures. The experiment is summarised in Table 2.9. 

This study is an example of a one-group pretest posttest design. (In fact, 
Cohen and Manion call it a pre-experiment.) Such studies have weak internal 
validity Ivcausc it is almost impt)ssiblc to state with any confidence that the 
dependent variables, that is, postintervention improvements in reading, were 
due to the independent variable, that is, the book 'flood'. Many other reasons 
tor improvements could be advanced, including the effect of taking part in an 
educaiional innovation, the fact that the children matured during the course 
of the experiment, the effect of other variables which have not been con- 
trolled, and so on. 

The internal validity of this study would have been greatly improved by 
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TABLH i.9 SUMMARY OF NEW ZEALAND BOOK FLOOD EXPERIMENT 



Rationale 


To examine p>os$ibIe correlations between extensive 




reading/ wide availability of books and reading 




habits and skills 


Experimental subjects 


Maori children in two schools 


Control subjects 


None 


Independent variable 


Wide access to books 


Dependent variables 


Scores on reading tests 




Qualitative infortnation on reading habits 


Outcome 


Significant improvement in reading scores 


TABLE Z.IO CONTRASTING PRE-EXPERIMENTS, QUASI-EXPERIMENTS, TRUE 


EXPERIMENTS 




Type 


Characteristics 


Pre-experimcnt 


May have pre- and posttreatment tests, but lacks a control 



group 

Quasi-experimcnt Has both pre- and posttests and experimental and control 

groups, but no random assignment of subjects 

True experiment Has both pre- and posttests, experiinental and control 

grouixs, and random assignment of subjects 



the addition of one or more control groups. Given the fact that it was not 
feasible to assign subjects at random to experimental and control schools, it 
would be important for the researchers to match the schools in terms of size, 
background, organisation, teaching methods, and background abilities of 
learners. The research design would also be strengthened by providing sub- 
jcas with carefully selected pr:- and posttreatment tests. Finally, it would be 
highly desirable to collect qual; lative data on reading habits, classroom teach- 
ing procedures, and so on, in both experimental and control classrooms. Such 
qualitative information is often crucial for the interpretation of quantitative 
data, as Spada (1990) has shown. The major differences between prc-cxperi- 
ments, quasi-experiments, and true experiments are set out in Table 2.10. 

The psychometric study: an example 

In this seaion, I shall illustrate the characteristics of psychometric research 
S cjking you in some detail through a research report based on such a study. 
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The study is an investigation by Chaudron and Richafds(1986) into the eftcct 
of discourse markers on the comprehension of lectures by non-native speak- 
inj; university students. I have chosen this study because it exemplifies some 
of the central points which have been discussed in the chapter. 

RESEARCH AREA 

The general research area investigated by Chaudron and Richards was the 
comprehension of university lectures by non-Fnglish speaking background 
students. 

AIM 

The aim of the study was to investigate the effects of discourse signals and 
markers on the comprehension of lectures by second langauge learners. 

questions / HYPOTHESES 

Two research questions were posed by the researchers: 

1. What is the effect on L2 learners' comprehension of lectures of the use of 
discourse markers which indicate the overall organisation of lectures - 
that is, macro markers, which signal the macro structure of a lecture 
through highlighting the major information in the lecture and the 
sequencing or importance of that information.^ 

2. What is the effect on L2 learners' comprehension of lectures of the use of 
micro markers, which indicate links between sentences within the lecture 
or function as fillers.' 

The.se two questions were formulated as three research hypotheses: 

Hypothesis 1: L2 learners would comprehend a lecture better when micro 
markers were added than when no markers were added. 

Hypothesis 2: L2 learners would comprehend the lecture with macro markers 
better than the lecture with only micro markets. 

Hypothesis 3: L2 learners would comprehend best the lecture with both micro 
and macro markers. 

In other words, the resea rchers were suggesting ( 1 ) that the comprehension 
of lectures would be enhanced if the lecturer used different types of signals; 
(2) that lectures in which both the overall structure and organisation, as well 
as between-sentence links, are explicitly signalled will be more comprehen- 
sible than lectures in which there are no signals, or in which only one type of 
signal is present, and (3) that signalling overall structure and organisation will 
result in greater comprehension than signalling relationships between sen- 
tences only. 
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ll^STIKK ATION (WIIY T\i\ ()HISTION IS WDKTll 
IS VK Sl ICA I INC.) 

I he researchers justiHed the research on the grounds that, with greater num- 
ber, of non-native speakers of F.nglish entering universities .irouiui ilie world 
m which English is the medium of instruction, there is .i need lor resc.uch 
into the processes underlying performance in a second or foreign i.uigu.ige 
\Mthin a university environment. Such research on pri)vido gukl;iiKv for 
te.icher education, instructional materials, and curriciihini dovelo|jincni. 

In their literature review, the researchers contrast IvKtoni-up and t()p-dt)wn 
a.mprehension strategies. Hottom-up strategics kgin with the processing ot 
uulividiial sounds, then words, then clauses, and then whole nicss.iges. I op- 
down strategies, on the other hand, K'gin with overall knowlalge of the sub- 
ject matter and structure of the text to be comprehended. For many years 
there has been an ongoing debate al-H)ut the relative merits of lop down .uui 
K)tiom-up approaches to language processing. While the debate has dealt 
prmcipally with reading comprehension, it has also Ix-en important in the lit- 
erature on listening (see, for example, Anderson and l ynch mH; Host 1990). 
The debate, and the research into comprehension processes on winch it rests, 
IN important for pedagogy, because the different approaches have clear imijh- 
cations for the design of teaching strategies and classroom materials. 1 helx)t- 
tom-up approach suggests that we begin by teaching the individual elements 
which constitute language (that is, phonemes and graphemes), then words 
and phrases, then sentences, and finally whole texts. The top-down apjjroach, 
on the other hand, suggests that we U-gin with whole texts and encourage 
learners ro use their knowledge of text structure and tiie ovcr.iil purpose of 
the speaker/ writer to orient them to the text. In recent years, it has been siig- 
gc-bted that both bottom-up and top-d(^wn strategies are iuiijortant, ami that 
the g(x>d reader/ listener is able to einpioy both appropriately. 

Chaudron and Richards argue that :x)th top-down and l-H)it()m-up pro- 
cessing are important in understanding 'ccturcs, and they cite studies which 
suggest that both macro markers and m-:ro markers can facilitate this under- 
standing. Macro markers can help learners with the overall organisation of a 
lecture, as well as a.ssist in comprehercing the functional intention of the 
speaker in relation to subsidiary parrs ; ; the lecture. Micro markers, which 
itrve as breathing spaces and which provide learners with opportunities to 
employ bottom-up processing strateg-es, should also facilitate the compre- 
hension of lectures. 

SUBJECTS 

There were rwo groups of subjects in tiic study: a pre university group con- 
sisting of 71 ESI. (English as a second l^mguage) students enrolletl in a iini- 
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vcrsity language program, and a university group, consisting of 81 ESL stu- 
dents enrolled in university programs. The researchers had placement test and 
standardised test scores on the subjects. 

MATERIALS 

Hour versions of a lecture on American history were used in the study. The 
baseline version was a 'natural' version presented to ESL students ('natural' 
m that It was not altered by the researchers). A macro version contained 
markers of discourse organisation such as the following: 

what I'm going to talk about today . . . 
let's go back to the beginning . . . 
this brought alx)ut new prolilenis . . . 

The micro version contained 'filler' phrases and markers of intcrsentential 
relations such as the following: 

tciiipornl links: then, and, now, after this, at that time 

ciiusol links: Ix'causc, so 

caiitnisfifc relationships: but, actually 

rchitii'c emphasis: you see, unbelievably, of course 

framing/segmentation: well, OK, all right.' 

The final version combined versions two and three, and contained both macro 
and micro markers. The following extracts, taken from the study, show 

examples of all four versions. 

Ihfeliiie rrrsion: 

•The United Stntcscimc into existence oftci.illy in 1783 after eight years of war 

"Ky 1 S(U, tliL- ()rii^in.il thirteen colonics had doubled in size . . ." 

Micro ivrsion: 

"Well, the United States came into existence officially in 1783 after eight years of 
w.ir ..." 

"Anil so, by I8(M, the original thirteen eolonic-s h.id doubled in size . . ." 

Afi;( ro tvrsion: 

"To Ivgin with, the United States came into existence officially in 1783 after eight 
years of war ..." 

"Wh,,t we've ccine to by now was that by 1803, the original thirteen colonies had 

Joiinlecl in size . . ." 

Wicro-ttutcro version: 

"Well, to Ix-pin with, the United .St.ites cinic into existence officially in 1783 after 
eight yc.irs ot w.ir . . ." 
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"And so, what we've come to by now was that by 1803, the original thirteen 
colonies had doubled in size . . ." 

PROCEDURE 

The four different versions of the lecture were assigned at random to the sub- 
jects, who were provided with a set of instructions on how to do cloze, mul- 
tiple choice, and true/ false questions. They were also given two short practice 
listening passages. 

Dependent variable: Scores on cloze, multiple choice, and true/false 
questions. 

Independent variable: Micro and macro markers in academic lectures. 
TYPE OF DATA 

Subjects' scores on the cloze, multiple choice, and true/false tests. Subjects' 
placement scores. 

(Before proceeding, you might like to pause and think about the statistical 
tools the experimenters might use to analyse their data. Are the researchers 
interested in comparing means, in comparing frequencies, or in investigating 
the extent to which two or more events correlate?) 

TYPE OF ANALYSIS 

The data were analysed using the following statistical procedures: Pearson 
product-moment correlations were calculated between the listening compre- 
hension and the various placement test scores. (As we saw earlier, correlation 
is used to test the strength of association between sets of scores - in this 
instance, the strength of association between listening comprehension and 
placement scores. In essence, the researchers were asking, 'Does a student who 
scored well on the placement test also score well on the listening 
comprehension?') 

Possible differences in the mean scores on the different versions of the lec- 
ture were tested by means of an analysis of covariance (ANCOV A). (This per- 
forms a similar analytical task as analysis of variance, but takes into account 
differences in the learners' proficiency levels.) 

CONCLUSIONS 

The researchers found that macro markers (that is, the higher-order discourse 
markers which indicate the overall organisation of a text) were more con- 
ducive to successful recall of the lecture than micro markers (that is, the 
lower-order markers of segmentation and intersentential markers). They 
point out that this is in line with the top-down theory of comprehension. 
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which suggests that explicitly signalled guidance on the organisation of major 
ideas in the discourse will help listeners construct appropriate schematic mod- 
els of the lecture. 

Another finding was that the micro markers did not facilitate comprehen- 
sion. They account for this by suggesting that these markers probably do not 
add enough content to make the subsequent information more comprehen- 
sible, that the markers may actually make the lecture less comprehensible by 
distracting the listeners, and that the baseline version may already have been 
slow enough to allow learners to obtain the maximum comprehensible input 
without the aid of further pauses and built-in redundancy. 

Another surprising finding was that the combined micro-macro version 
was actually inferior to the macro version. The researchers attempt to 
account for this finding by suggesting that the insertion of the micro markers 
mcreased the listening load on the subjects without adding information 
which might have facilitated their overall comprehension, and that this 
detracted from the effect of the macro markers alone. 

CRITIQUE 

This Study is a carefully conceived and conducted investigation into the lis- 
tening comprehension of NESB university and pre-university students which 
illustrates some important characteristics of the experimental method. The 
researchers derive a number of questions/hypotheses from their review of the 
empirical and theoretical literature on the subject at hand. They suggest rela- 
tionships between variables, and test these out through the exposure of ran- 
domly assigned subjects to control and experimental groups. The data yielded 
by the experiment are then subjected to statistical analysis, which enables the 
researchers to assess the probability that the independent variable did have an 
effect on the dependent variable. 

The study is particularly interesting to students of research methods, 
because the researchers do not attempt to gloss over or minimize the practical 
difficulties which emerged in the course of the study, nor the interpretive 
problems associated with the unexpected outcomes of the study. The fact that 
placement and standardised test data were used, rather than a listening pre- 
test, means that we should probably view this study as a quasi-experimenr 
rather than a true experiment. In their paper, the researchers acknowledge 
the complications that the use of these test data occasioned - for example 
some of the subjects were tested several months after the others, which made 
their pretest scores highly suspect, and these subjects were subsequently 
excluded from the research. Ideally, the researchers should have constructed 
a test to operationalise the construct 'listening comprehension' and used this 
test, or versions of it, for both pre- and posttesting purposes. 

I have already discussed the importance of constructs in research and the 
need for researchers to operationalise their constructs. As we have seen, con- 
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btructs are unobservable qualities which are assumed to uiulerlie obsi-rv.ible 
behaviour. Operationalisiiig a construct means Jctiiiiii}; ir m rerms of observ- 
able behaviour. In the case of C-'haudron and Richaitis, the key construct is 
"listening comprehension', and the researchers operationalise the constriici in 
terms of the ability ro answer true/lalse, multiple choice, ;iiid clo/c iiiicstions 
aK)ut the listening text. While the use of such rests has been criticised as in.ui- 
equate as a measure of overall coniprehension (see, for example, Aslanian 
I9S5), this type of experimental rese.ircli must necessarily rest on such mea- 
sures. In defence of Chaudron and Richards, ir should he pointed our tii.u 
they attempted to strengthen the internal validity of the researcii by using 
not one but three measures of com[)reheiision. The iiiterii.il valiJii y may li.n e 
Ix-en srreiigthened further by supplementing rhe qu.iniii.uive d.ita wiili 
itative data, such as follow-up interview data of the type employeil by 
Aslanian. 

The other noteworthy aspect of the study is rhe way in which the research- 
ers dealt with the unexpected finding that the combined micro-macro version 
was inferior to the macro version. The researchers considered and dismissetl 
the notion that the groups hearing these passages were less proficient, and also 
the possibility that the quality of the recording for that particular vei sioii was 
inferior. The only explanation they coiikl come up wiili was that adding liie 
micro markers increased the listeners' attention requirements without adding 
semantic information, and that this detracted from the efteci of the macro 
markers. While this explanation may not Ix; pariicul.irly convincing, it is rhe 
researchers' best guess at the probable cause. Research, even when carrieil out 
in controlled settings, often results in some outcomes which are counter-intu- 
itive in the light of other outcomes and therefore difficult to account for. This 
is one such finding. It underlines the fact that experimental researchers, no 
less than those engaged in naturalistic, interpretive inquiry, have lo interpret 
and account for their data. With hindsight, it might have Ix-en wise to con- 
duct a pilot study. Such a stuiiy could have revealed some of ihe prohlcnis 
which emerged (such as problems wirh the micro version), and thereby ena- 
bled the researchers to strengthen their major investigation. 

Conclusion 

The formal experiment and its variants, the pre-experinieiu and qiiasi expcr- 
iment, are important research tools in language study, and they have addeil 
significantly to our knowledge of language learning, teaching, and use. 
Experiments are designed to collect .iata in such a way that threats to the 
reliability and validity of the research are minimised. Kxperimental rescarcii- 
ersare particularly concerned with ti v issue of external validity, and ihe for- 
mal experiment is specifically designc. ro enable the researcher to extra polaie 
the outcomes of the research from the sample to the broader population. In 
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order to appreciate the reasoning which lies behind such extrapolations it is 

necessary to understand the logic of inferential statistics and some of the more 
commonly employed statistical tools. These have been the focus of this 
chapter. 

In the chapters to come, we shall see that experiments are by no means the 
only tool nvailahic to the applied linguistic researcher, and that for some types 
of research, the tool is inappropriate. In the next chapter, we turn to ethnog- 
raphy, which is based on a very different logic from the formal experiment, 
.ind which seeks to answer quite different questions about the nature of lan- 
guage lenrning and use. These two methods, experiments and ethnography, 
form extreme points on a methodological continuum, and so they will pro- 
vide us with .several signposts from which we can take our bearings in the rest 
of the book. 



Questions and tasks 

1. Write out your own definitions of the following terms: construct, fre- 
quency t.nble, central tendency, variable, variability, mean, variance, standard 
deviation, population, s.imple. 

2. Im.igine that you are about to carry out a detailed investigation of the 
foreign reading skills of a group of high school (or secondary school) students. 
I he school h.xs provided you with a great deal of information on each stu- 
dent, including the following. For each of the variables listed identify the 
most likely type of scale it would represent. 

Potential variable Most likely type of scale 

a. Sex 

b. Age 

c. Class rank 

d. Nationality 

e. First language 

f. Scores on proficiency test 

g. Rank on class test 

h. Amt. of time spent studying 

l-'rencli 

i. Whether parents are native 

speakers of the language 

3. Which statistical procedure would you employ for the following situa- 
tions? Indicate the appropriate procedure by placing a number in the bracket. 

1 - /-test (tests differences between two means) 

2 - ANOVA (tests differences between multiple means, and tests for pos- 

sible interaction between means) 
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3 - correlation (tests strength of association between variables) 

4 - chi-square (tests association between frequencies) 

a. This study investigated the relationship between scores on a traditional 
standardised test of oral proficiency and a new test of communicative 
speaking ability. The investigators wanted to determine whether subjects 
who scored well on the traditional test did well on the innovative test and 
vice versa. [ J 

b. This study examined the effect of content familiarity on the reading com- 
prehension of secondary level second language learners. Subjects were 
given two test passages to read, and their comprehension was tested on a 
multiple choice test. The test passages were analysed using the Lix read- 
ability formula, which showed they were both at the same level of lin- 
guistic difficulty. However, one passage was on a topic familiar to the stu- 
dents while the other was on an unfamiliar topic. [ J 

c. In this investigation, the researcher wanted to find out whether the native- 
like production of word-final consonants and consonant clusters varied 
according to the type of communicative task in which the learner is 
engaged. She collected data from a single subject as he took part in three 
different tasks - free conversation, oral reading of continuous text, and 
elicited imitation of words and short phrases. [ ] 

d. A study was established to test the hypothesis that field-dependent learners 
would benefit more from learning grammar inductively, while field-inde- 
pendent learners would benefit from learning deductively. One group of 
field-dependent subjects and one group of field-independent subjects were 
taught inductively. A second group of field-dependent subjects and a sec- 
ond group of field-independent subjects were taught deductively. Pre- and 
posttreatment scores on a standardised test of grammar were adminis- 
tered. [ J 

4. The following hypothetical study is presented by Brown (1988: 18-19). 
Read the study and identify the independent and dependent variables. 

Jas Menbrow noticed that people do not always perform in their native languages 
with equal ability. It (Kcurred to him that there might be some relationship between 
the native language ability of his American students and their proficiency in French 
after three years of high school study. To investigate this possibility, he 
administered the verbal subtest of the Scholastic Aptitude Test (to measure 
proficiency in native English) and his school district's French Proficiency Test to all 
132 students who were finishing their third year of study. He was concerned that 
differences between the sexes should not interfere with the results, so he eliminated 
all males from his analysis. Thus 89 female students were left. In addition, he 
wanted to determine what differences there might be between the women in the 
college-bound track (44 students) and those in the accelerated track (45 students). 
So he considered the two groups separately on this basis. 
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5. The effect of randomisarion can be demonstrated by carrying out the fol- 
lowing experiment devised by I'illiner and reported by ('ohen and Manion 
(I9K5: 191). 

Select twenty cards from a pack, ten red and ten black. Shuftle and deal into two 
ten-card piles. Now count the number of red cards and black cards in either pile 
and record the results. Repeat the wht)le sequence many times, recording the results 
each time. You will s(X)n convince yourself that the most likely distribution of reds 
and blacks in a pile is five in each; the next most likely, six red (or bi.ick) and four 
black (or red); and so on. You will be lucky (or unlucky for the purposes of the 
demonstration!) to achieve one pile of red and the other entirely of black cards. The 
probability of this happening is 1 in 92,378! On the other hand, the probability of 
obtaining a 'mix' not more than 6 of one colour and 4 of the other is about 82 in 
l(X). 

If you imagine the red cards to stand for the 'better' ten children and the black 
cards to stand for the 'p<K)rer' ten children in j class of twenty, you will tonchulc 
that the operation of the laws of chance alone will almost probably give you close 
equivalent 'mixes' of "better' and 'pot)rer' children in the experimental and control 
groups. 

Replicate this experiment yourself. 

Look at the reported conclusion of this experiment (the last line in the pre- 
ceding extract). Based on your experiment, do you concur with this 
conclusion.' 

6. Examine the following research summary adapted from Cohen and Man- 
ion (1985: 172). What type of study is it? What criticisms, if any, would you 
make of it? 

Out of 802 students who began their studies at the University of Bradford in 1966, 
102 (lrop[x:d out at the end of their first year. On entry to the university, the entire 
freshman intake had provided academic and personal information about their 
backgrounds, their interests, their motivations and their values. 

In an investigation of factors associated with failure at university, comparisons 
were made between dropouts and non-dropouts. 

In line with previous studies, university failure was found to relate (!) to inferior 
educational qualifications on entry; (2) to less certainty about choice of career; (3) to 
a greater degree of worry over abilities to pursue a university course of study; and 
(4) to feelings of being overwhelmed by the academic work demanded. 

7. Design one of the following: (Da follow-up investigation to the Chaudron 
and Richards study, or (b) an alternative to the study (this can either be an 
alternative way of investigating the phenomenon of discourse markers and 

comprehension or an investigation of some other aspect of language learning 
and use which employs a similar research design). Include the following: 

question /hypothesis subjects 

elicitation procedure type of data 

type of analysis anticipated problems/difficuiries 
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Further reading 

A number of books on statistics in linguistics and applied linguistics have 
appeared in recent years. The best of these is by |. D. Brown (1988), who jjro- 
vides an intrcxluction to the subject which is generally sensible and rcMclaiile, 
although there are several conceptual leaps in the b(K)k that couki make it 
difficult going, at times, for the naive reader. A detailed treatment of stai istics 
and experimental research, as well as an introduction to general principles of 
research, is to be found in Hatch and Lazaraton (1991). 

For those with no knowledge of linguistics, Rowntrec (1981) provides a 
useful introduction. Although this book is not written sfx-cificaily for the lan- 
guage teacher or applied linguist, it is, like most of Rowntree'sb<K)ks, sensible, 
easy to read, and witty. Another excellent publication is Robson (1973). This 
provides 'recipes' for calculating various statistics by haml. 
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l-iliicuors shoiilJ not l.iok lure for experiments, controlled conditions, nnd 
sysu niatic score kecpinn »" tf'c ncndcmic gains and losses of specific children Nor 
should psychohnguisls look here for data taped nt periodic intervals under similar 
conditions over a predcsiRnated period of time. Wlwt this ly«,k docs ilo is record 
the natural flow of comnuinity ami classroom life over nearly a decade. Tfic 
dcscrr[M,ons Iktc | arc] of iIu- actual processes, activities, and attitudes involved in 
iIk- cncultiiratioM of children. 

(Shirley Brice Heath, 198.^, Ways with Wnrds) 

l-lhunfrrcphy contmsts nmrkcilly v\'ith tiic experimental method in its 
;isMmiptions, nictliods, and attitudes to evidence. In principle there is no rea- 
son why research profjrams should not integrate psychometric and ethno- 
graphic methods of investigation, and, indeed, several calls have already been 
made for such integration. In practice, however, integrated approaches seem 
to he almost non existent. I have already argued that this reflects the fact that 
tlie two approaches arc iniderpinned by very different conceptions of social 
reality - truth and the nature of evidence. It also reflects the fact that both 
traihtions are seeking to answer different types of questions. 

The ilifferent conceptions of 'truth', 'reality' and evidence' held by some 
language researchers is one reason for the growing attention being paid to tlie 
use of ethnographic fcciiniqiies for gathering and analysing language data 
Interest m hiulmg alternatives to formal experiments has also been stimulated 
by a scepticism over the ability of psyciiometry to 'produce the definitive 
answers that some researchers expect' (F.llis 1990a: 67). F.llis advances two 
reasons for this scepticism. In the first place, the relationship between instruc- 
tion and learning is extremely complex. It is not a linear relationship, and 
there is no one-to-one relationship k-twecn teaching and learning. Kxperi- 
mem.il research can therefore only provide us with an understanditig of itidi- 
vidiial pieces of the language learning jig.saw, but not the whole puzzle. Sec- 
ond y, according to llllis, classrooms do not exist simply to provide cannon- 
loilder for research. I lie relationship between fuidings from a formal exper- 
iment, eondiieteii under lalx)ratory conditions, and classroom practice is 
complex and indirect. 

Iniiov.ition in the classr<«.iii can never W just a question of iniplemcntin); a 
recommendation derived from research. If is .iKvays a process of negotiation 
involving ihc teaclur's overall educational ideology, the leamer's expectations and 
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preferences and local constraints that determine what is feasible. There is no single 
pedagogical solution which is applicable to all classrooms. (Ellis 1990a: 68) 

In this chapter, I shall define ethnography and describe its central character- 
istics and principles. I shall also look at the key research concepts of reliability 
and validity as they relate to ethnography, and will discuss the importance of 
context to ethnographic inquiry. In the final part of the chapter, I shall high- 
light some of the central concerns of this chapter and the preceding one, by 
contrasting psychometry and ethnography. The chapter seeks to address the 
following questions: 

- What do we mean by ethnography^ 

- What are the key principles guiding ethnographic research.' 

- How might one deal with threats to the reliability and validity of this type 
of research? 

- Why is context important to ethnographic research? 

- In what ways does ethnography contrast with psychometric research? 



Principles of ethnographic research J^,,^ Q)/ '^A/^' 

Ethnography has suffered somewhat from beingapplied gather loosely to any ' 
research that is not a formal experiment, giving rise, in some quarters, to the 
suspicion that the tradition and its practitioners lack rigour. However, as 
Chaudron (1988) and others have pointed out, true ethnography demands as 
much training, skill, and dedication as psychometric research. 

Wilson (1982) identifies the roots of ethnography in anthropology and soci- 
ology, although there is also a strong tradition in research into animal behav- 
iour (see, for example, Martin and Bateson 1986). Wilson relates the traditiori" f 
to two sets of hypotheses about human behaviour. These are the naturalistic- 
ecological hypothesis and the qualitative-phenomenological hypothesis. 

The naturalistic-ecological perspective has, as its central tenet, the belief 
that the context in which behaviour CKCurs has a significant influence on that 
behaviour. It follows that if we want to find out about behaviour, we need tq^ 
investigate it in the natural contexts in which it occurs, rather than in the 
experimental laboratory. Arguments in favour of field research as opp>osed to 
laboratory research are supported by studies of^particular phenomena_which 
come up with different findings according to whether the research is con- 
ducted in a laboratory or in the field. For example, Bellack, Hersen, and Tur- 
ner (1978) found that subjects performed in a role-play situation very differ- 
ently from the way they performed in real-life social situations where the 
same behaviours were observed. It has also been observed that parents and 
pupils resf)ond differently to questions according to whether they are po.sed 
in school or at home. 

It would seem to be a matter of commonsense that if one wants to gener- 
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Van Lier (1988) also identifies cultural description as a central cliar;icter- 
istic of ethnography. He presents two views on ethnography, the 'weak ' view 
and the 'strong' view. The weak view, which according to v.ui l.icr is cur- 
rently prevalent in applied linguistics, sees ethnography as essentially inlerior 
to psychometry, as it consists of unstructured and unsystematic observation. 
The principal virtue of ethnography, according to the weak view, is rh.ir it 
may throw up questions or hypK)theses which can subsetiueiitly he tested in a 
forrnal experiment. In other words, it is essentially a 'ground-clearing' oper- 
ation rather than a valid tradition in its own rigiit. The strong view, to which 
van Lier himself subscribes, sees ethnography as a valid research p.iradigni in 
its own right: . . ethnography is theory-building, and thus the core of a 
humanistic approach to social science. In this sense it can Iv traced back to 
naturalistic approaches to swial science . . .' (van l.ier 1488: 54). llliini,i(ely, 
whether one subscribes to a strong or weak interpretation ot ethnogr.ipiiy 
will defjend on one's views on the status of knowledge, the nature oi 'truth', 
and what one accepts as legitimate evidence. 

Despite differences of emphasis, these different statements ail agree that 
ethnography involves the study of the culture/characteristics ol a group in 
real-world rather than laboratory settings.\The researcher makes no artompr 
to isolate or manipulate the phenomena under investigation, and insights and " 
generalisations emerge from close contact with the data rather than from a 
theory of language learning and use. The principles are exemplified in an 
influential educational ethnographic study carried out in the sixties by Smith 
and Geoffrey (1968). The pnrpK)se of the study was to descrilv what goes on 
in the contemporary urban classroom. The data collection procedure lor tiiis 
study was deceptively simple - Smith, a university-based researcher who was 
trained as a psychologist, spent a term sitting in Geoffrey's classroom as a 
participant observer. His database consisted of extensive field notes of what 
he observed, as well as interviews with the teacher and students. These data 
were used to construct a rich descriptive and interpretive picture of the com- 
plexities of an urban classrm)ni in the sixties. 

In language teaching, a similar investigation was carried out by Freeman 
(1992). Freeman became a participant observer in a French as a foreign lan- 
guage classroom, and his database iiicluded lesson transcripts, fieldnotes, and 
interviews with the teacher and students. The analysis consists of discursive 
and interpretive work on the database. Freeman concludes from his investi- 
gation that: 

The process of evolving a shared understanding of what to Icaru and how to !e;irn it 
is at the heart of what makes [the teacher's] classes work, it takes place against the 
backdrop of constant social interaction . . . and is intimately tied to sliariiig 
authority and control. [The teacher] has lx.'en able to make public the process of 
creating and internalizing the language precisely becausf slit allows liie talk and 
activity in her class to be largely self-regulatcd. Stiidems come to coiurol tlieiiisclves 
in their interactions; that control goes hand-in-hand with authority over the 
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TABI.i; 3.1 CHARACTERISTICS OF ETHNOGRAPHIC RESEARCH 


Chariicleristic 


Gloss 


Contextual 


The research is carried out in the context in which the subjects 




normally live and work. 


Unobtrusive 


I'lic researcher avoids manipulating the phenomena under 




investigation. 


Longiruilinnl 


The research is relatively long-term. 


Collaborative 


The research involves the participation of stakeholders other 




than the researcher. 


interpretive 


I'hc researcher carries out interpretive analyses of the data. 


Org.iiiic 


There is interaction between questions/ hypotheses and data 




collcction/intcrpretntion. 



language. Both involve the rcsptinsibility to an inner sense of rightness for 
appropriate Ix'haviour and for accurate language use. I his responsibility is 
individual and collective. ( I be teacher] is a resource for the language and a source 
for criteria and explanations of correctness. Likewise she is the source of activity in 
the classrooni and a resource for successful accomplishment of that activity. 

These examples share .several characteristics. In the first place, the research 
takes place in context, with an attempt to miniinize the disruption caused by 
tlic researcher's intrusion. The researcher does not attempt to control or 
manipulate the phenomena under investigation. The research is relatively 
long-term, taking place over several weeks, months, or even years. It entails 
tiic coil:il'H)rntivc involvement of several participants, including the 
researcher, the teacher, and the learners. Finally, generalisations and hypoth- 
eses emerge during the cour.se of the data collection and interpretation, rather 
than Ix-ing predetermined by the researcher (these principles are revisited in 
the exteniicil oxamplc presented later in this chapter). Table 3.1 summarises 
these characteristics. 

The principles of ethnographic research differ in important ways from 
those of psychomctry, and result in very different research procedures. Eth- 
nography places great store on the collection and interpretation of data, and, 
in marked contrast with the experimental method, questions and hypotheses 
often cntcrgc during the course of the investigation, rather than beforehand. 
This is anathema to the proponents of experimental approaches to research. 
It is sometimes suggested that psychomctry is an hypothesis in search of data, 
whereas ethnography is data in search of an hyptnhcsis. While this is some- 
thing of an exaggeration, in that ethnographers often begin with questions, 
if not formal hypotheses, it does highlight one important characteristic ofeth- 
nvigraphy: the tact that there is often an interaction between questions and 
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data. During the course of their investigations, ethnographers may obtam 
data which do not support their original questions or hypotheses but are sug- 
gestive of others. They may therefore end up answering questions other than 
those with which they began, thus violating Brown's (1988) principle of log- 
icality. In fact, while it is considered unacceptable for psychomctricians to 
begin with one set of questions and conclude by answering others, the liter- 
ature is littered with experiments which do just this. Notwithstanding these 
violations, many of these studies provide important insights into language 
learning and use, which suggests that perhaps the principles of formal exper- 
imentation could well be revised. 

The practice of deriving theory from data rather than the other way round 
is known as grounded theory. This 'data first' approach is criticised by Long 
(1986) and Gregg (1989), who argue that a 'theory then data' rather than 
'data then theory' approach is more efficient and rational. This approach to 
research sits more comfortably with a weak view of ethnography and, indeed, 
in a later paper Long (1990) argues that the primary purpose of educational 
ethnography is to describe classroom processes so that they may later be sub- 
jected to experimental manipulation. 'It would be premature and unwar- 
ranted to make causal claims on the basis of descriptive studies or to offer 
anything more than hypotheses as to potential explanations of the findings 
(Long 1990: 7). While it is true that there are dangers in making strong causal 
claims on the basis of description, 1 do not believe that this leaves ethnography 
in a position which is subservient to experimentation. Ethnography is a valid 
tradition in its own right, and should not be considered simply as an hypoth- 
esis-generating device for experimental research. 

Numerous writers have set out principles to define and guide ethnographic 
research. LeCompte and Goctz (1928) argue that ethnography is defined by 
the use of participant and non-participant observation, a focus on natural set- 
tings use of the subjective views and belief systems of the participants m the 
research process to structure that research, and an avoidance by the investi- 
gator of manipulating the study variables. Watson-Gegeoand Ulichny (1988) 
identify several key principles of ethnographic research. These include the 
adoption of a grounded approach to data, the use of 'thick' explanation and 
going beyond description to analysis, interpretation, and explanation. They 
point out that ethnography involves interpretation, analysis, and explana- 
tion - not just description. 'Explanation takes the form of "grounded the- 
ory which, as wc have seen, is theory based in and derived from data, and 
arri'ved at through a systematic process of induction' (p. 76). (The most com- 
plete treatment of grounded theory is to be found in Glaser and Strauss 1967.) 
Their two other key principles are 'holism' and 'thick explanation. Holistic 
research must take into account both the behaviour of the individuals and/ 
or groups under investigation and the context in which the behaviour occurs, 
which has a major influence on the behaviour. There are two dimensions to 
this type of analysis, a horizontal dimension and a vertical dimension. The 
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horizontal, or historical, dimension refers to the description of events and 
behaviours as they evolve over time. The vertical dimension refers to the fac- 
tors which influence behaviours and interactions at the time at which they 
occur. The principle of 'thick' explanation refers to the importance of taking 
into account all of the factors which may have an effect on the phenomena 
under investigation. Of course, deciding what is or is not relevant and salient 
is a subjective and reiativistic matter, which is why ethnographers generally 
insist on 'thick' description, that is, the collection of data on all of the factors 
which might impinge on the phenomena under investigation. 

The need to go beyond description to explanation and analysis is taken up 
by Goodson and Walker (1983), who argue that educational research should 
focus strongly on 'portrayal', even if this is at the expense of analysis and 
explanation. They point out that the choice of a research methodology is 
more than a technical issue: It is political and moral as well, and they propose 
a new research genre which is separate from either ethnography or case study, 
which they call 'story telling'. While admittinganxiety at the threats to objec- 
tivity, reliability, and validity posed by their approach, they claim that rich, 
descriptive accounts seem to offer 'a kind of intermediate technology of 
research adapted to the study of practical problems in realistic time scales 
without the prospect of ten years' initiation among dwindling (and probably 
best left) tribes of Primitives' (p. 29). We shall look in greater detail at their 
proposals in the next chapter, when we consider case study methodology. 

As we saw in Chapter 1, Chaudron (1988) identifies ethnography as one of 
the four major traditions in applied linguistic research, although he does not 
devote a great deal of his book to research carried out within this tradition. 
He characterises ethnographic research as a qualitative, process-oriented 
approach to the investigation of interaction, and points out that it is a rig- 
orous tradition in its own right, involving 'considerable training, continuous 
record keeping, extensive participatory involvement of the researcher in 
the classroom, and careful interpretation of the usually multifaceted data' 
(p. 46). 

The reliability and validity of ethnography 

The major criticisms leveled at ethnography by proponents of quantitative 
research concern the reliability and validity of such research. Most of these 
criticisms stem from the fact that ethnographies are based on the detailed 
description and analysis of a particular context or situation. Because of the 
quantity of data yielded in these studies, it is generally impossible to include 
anything but a small amount of the data in a published account of the 
research. This makes it difficult for outsiders either to analyse the data them- 
selves (and thereby establish the internal reliability of the study) or to replicate 
the study (thereby establishing its external reliability). Whether or not inter- 
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nal and external validity are problematic will di-fx-mi on the scope ot ilu- 
research, and the researcher's purposes. If the researcher is not attcmptmg to 
establish a causal relationship between variables, the issue oi internal validity 
will be less problematic than if such a relationship is being sought. 1 he essen- 
tial consideration in relation to external validity is to what extent the hiulmgs 
from a study carried out in a particular site can be generalised to other sites. 
If the researcher is not concerned with the issue of generalisation, then the 
issue of external validity is not a concern. A detailed and considered analysis 
of problems associated with reliability and validity in ethnographic research 
is provided by LeCompte and Goetz (1982). They use 'ethnography as a 
shorthand term to encomp.iss a range of qualitative metliods, including case 
study research, field research, and anthro[K)logical research. 

LeCompte and Goetz deal firstly with reliability, which they dchne ;is the 
extent to which studies can be replicated. They argue that at first b ush eth- 
nography may 'baffle attempts at replication', in comparison with lalx)ratory 
experiments. Given the naturalistic setting, the fact that the researcher in.iy 
be attempting to record processes of change over time, and the possible 
uniqueness of the situation and setting, the use of standardised controls may 
be impossible. In reporting the research, constraints of time and space may 
preclude the presentation of data in a way which woiilil enable other 
researchers to reanalyse these and draw similar conclusions. 

External reliability, that is, the replication of the research by other.s, can l>e 
enhanced if the ethnographer is explicit about five key aspects of the research 
These are: the status of the researcher, the choice of informants, the social 
situations and conditions, the analytic constructsand premises, and the meth- 
ods of data collection and analysis. 

Attending to researcher status position requires researchers to Ix- ex[)licit 
about the social position they hold within the group king investigated. 
LeCompte and Goetz make the point that in one sense, no ethnographer can 
exactly replicate the finding of another because, even if an exactly parallel 
context could be found, the second researcher is unlikely to hold exactly the 
same status in the second scKial situation. A related problem is that oi finding 
parallel informants in the second research context. This is a potential problein 
of major proportions when it is considered that 'the extent to which know - 
edge is gathered is a function of who gives it' (LeCompte and Goetz 1 982: 
It is therefore imperative for researchers to descrilx- their informants 
extremely carefully. The social situation and conditions which obtain also 
need to be described explicitly. (LeCompte and Goetz document a study of 
education in an ethnic neighbourhood in which parents provided discrepant 
information in the school setting and in the neighbourhood setting.) Explicit 
definitions of constructs and premises are also crucial. 

Even if a researcher reconstructs the relationships and duplicates the inforni.uits 
and social contexts of a prior smdy, replication may remain impossible if the 
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constructs, definitions, or units of analysis which informed the originnl research are 
.du,syncrat,c or po<,rly dchncnted. Replication rcqw.rcs explicit identification of the 
Tlyl''" 'hnt underly choice of tcrn^inoiogy and methods of 

Finally, cthnoKrnplicrs need to present their methods so explicitly that their 
report cnn be used as a procedural manual by those wishing to replicate the 
rcsciircn. ° k >-<"»- mc 

A study has internal reiinbility if independent researchers, on anniysinK the 
pr-mary data, come to ti,c same concit.sions as the original investigators 
Many of the sa egunrds identified in relation to external reliability afe Z- 
t.ncnt here the key difference being that external reliability refers to the rep- 
lication of the original study, while internal reliability concerns the reanalysis 
of the oriRinal data by independent researchers. Such reanalysis is compli- 
cated by titc fnct that ethnographers rarely use standardised instruments such 
as observation schedules. Strategies identified by LeCompte and Goetz for 
guarding against threats to internal reliability include the useof low inference 
dcscnprors, nu, t.ple researchers, participant researchers, peer exannnation, 
aiu) nicchamcally recorded data. 

LOW INI-I-RKNCr. DKSCRIPTORS 

A low inference descriptor describes behaviour on which it is easy for inde- 
pendent observers to agree. For example in classrooms, wait time' (the time 
a teacher waits k twcen asking a question and then following up, for example 
by answering the question) and use of factual questions would be examples 
of low inference k^haviours. High inference behaviours, on the other hand 
are those requiring the observer to make inferences alxjut the observed behav- 
iour. A high inference descriptor would be 'student lacks interest in activity' 
kcausc It requires the observer to infer unobservable mental states from 
observ.-ible behaviour. The problem here is that it is the high inference behav- 
iours that are often of most interest. 

MLIITII-U.; RKSHAKCHFRS/ PARTICIPANT R K S E A R C H I! R S 

Tlic most effective way of guarding against threats to internal reliability is 
tiirough the useof multiple researchers. In much research this is not feasible 
k-c.uise a research team consisting of several mcmlx-rs can be extremely 
expensive particularly given the extended nature of much ethnographic 
rcscHch. An alternative is to enlist the aid of local informants to validate the 
interpretations of the ethnographer. For example, an ethnographic study of 
a school could involve teachers in reviewing and validating the researcher's 
data and conclusions. 
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TABLE 3.2 GUARDING AGAINST THREATS TO THE RELIABILITY OF 
ETH.nKiRAPHIC RESEARCH 

Type Questions 

Internal reliability Does the research utilise low inference descriptors? 

Does it employ more than one researcher/collaborator? 
Does the researcher invite peer examination or cross-site 
corroboration? 

Are data mechanically recorded? 

External reliability Is the status of the researcher made explicit? 

Does the researcher provide a detailed description of 
subjects? 

Does the researcher provide a derailed description of the 
context and conditions under which the research was 
carried out? 

Are constructs and premises explicitly defined? 

Are data collection and analysis methods presented in 

detail? 



PEER EXAMINATION 

Peer examination involves the corroboration by other researchers working in 
similar settings. According to LeCompte and Goetz, this can proceed in three 
ways. Firstly, researchers may utilise outcomes and findings from other field- 
workers in their report. Secondly, findings from studies carried out concur- 
rently may be integrated into the report. This provides a form of cross-site 
validation. Finally, provided sufficient primary data are included in the pub- 
lished report, these may be used for reanalysis by the researcher's colleagues. 

MECHANICALLY RECORDED DATA 

The final strategy researchers can employ to guard against threats in internal 
reliability is the use of mechanically recorded data (for example, in the form 
of audio or video recordings). This strategy allows for the preservation of the 
primary data. However, it must be remembered that these devices do not pre- 
serve all of the data, but only those data selected by the researcher for 
preservation. 

The five key strategies which I have just outlined, and which are designed to 
protect a study from threats to internal reliability, are summarised as a series 
of questions in Table 3.2. 

While the foregoing strategies are valuable, they may no* all be practical 
for someone carrying out an ethnographic investigation with limited time 
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and resources. The use of multiple sites and researchers in particular could 
prove to be extremely expensive. As I have already indicated the u e oflow 
nference descr.ptors .s also a problem, because high mference behav.ou^ nd 
constructs are often of greatest interest. (Ethnographies in languaL Ss 
rooms have mvest.gated phenomena such as motivation, mteres 'powe 
authonty, and control, all of which are high inference behaviours.) Throlr-' 
all thrust of LeCompte and Goetz's suggestions for guarding again t threats 

ally whaJ'th"" ^ '"u^"° ""^'^^ ' " ^ explfcitnl S- 

cally, what they are suggestmg .s that if one is careful in the collection and 
analysis of one s data, and if one is explicit about the way the data were ^ol 
mvi'gTtt:"^'^^^'' ""^ ^'^ re.iabili:"frr's 

I shall now look at problems associated with the internal and exrern.l 
vahduy of ethnographic research, once again follower e ana" " o 
LeCompte and Goetz, who argue that internal validity is one of the st ength 
of ethnographic research (a view questioned by a number of otherlsearche^ 
Txt nt'r' validity, you will recall, relates o £ 

extent to which an mvest.gat.on is actually measuring wha it purports to 

extend" n rSe" h"'"^'''' ^^'^ 'J-"--' T^'h t 

extent can research outcomes be extended to other groups.^ Dealing with 

^S^^uMot^^: TT ""^'T ^""'"S participants and 

other revirrh rl«:i„L tl j JDstractly than instruments used n 

experience of participants more accurately than do contr ed itti^as n 

For the researcher wishing to generalise beyond the context in which the d^r. 
reduce .ts comparability. The researcher can guard against this threat by 
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TABLE 3.3 GUARDING AGAINST THRKATS TO Till- VALIDITY 0\- I- 1 1 INOGR Al'l llc: 
RESEARCH 

Type Questions 

Internal validity Is it likely that maturational changes occurring during ilic 
course of the research will affect outcomes? 
Is there bias in the selection of inforinaius? 
Is the growth or attrition of inforniaius over time likely 10 attcci 
outcomes.' 

Have alternative explanations for phenomena been rigorously 
examined and excluded.' 

External validity Are some phenomena unique to a particular group or site and 
therefore non-coniparable? 

Are outcomes due in part to the presence of the researcl)cr? 
Are cross-group comparisons invalidated by unique hisiorieai 
experiences of particular groups? 

To what extent are al^stract terms and coiisiructs shared across 
different groups and research sites? 



describing phenomena so explicitly that they can be compared with other 
studies, or by carrying our multiple-site investigations. LeCompte and Goetz 
argue that the credibility of cross-group comparisons can be affected by four 
particular factors: selection effects, setting effects, history effects, and c<»n- 
struct effects. 

Selection effects can have a bearing on the external validity of an ethnog- 
raphy if the constructs under investigation are specific to a single group, or 
because there has been a mismatch between the chosen group and the con- 
structs for investigation. In discussing setting effects, LeCompte and Goetz 
point out that the very act of investigating a group, culture, or setting may 
have an effect which renders cross-group comparisons invalid. Their discus- 
sion brings to mind Labov s (1972) 'observer's paradox' (I.abov pointed our 
that the aim of sociolinguistic research is to find out how people behave when 
they are not being systematically observed, but the data can be obtained only 
through systematic observation). Cross-group comparisons may also be ren- 
dered invalid by the unique historical experiences of groups and cultures. 
Finally, construct effects can refer either to the extent to which abstract terms 
and concepts are shared across different populations, or to the extent to 
which explanations are regarded as valid across groups. A summary of these 
points is provided in l abic 3.3. 

In this section I have drawn principally upon the work of LeCompte and 
Goetz to address the various factors which may threaten the internal and 
external reliability and validity of ethnographic research. While the adoption 
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of an cthnogrnph.c approach to research can pose formidable problems of reli- 
ab,hty and vahd.ry, tliese problems are nor insurmountable'^ and Z a e 
praccal steps one can take to guard against them, although these may 
.nvolve addmonal time and resources. In the next section, I shall coTtextu' 
al.se and ,ll„stratc the operation of these factors with reference to a major 
mvcstigatiof, of language learning and use. ' 



The importance of context in ethnographic inquiry 

In this section, I shall attempt to expand on and exemplify some of the prin- 
Sh ^^^T^^^P'^l'by analysing a major study carried ou*^ by 

Heath (1983 . I have chosen this study because it is arguably the major con 
tcmp,,rary ethnograpluc investigation of language development to have been 
published smcc the early sixties. ""-vcuccn 

RESEARCH AREA 

The acquisitio.i of a first language at school and at home. 

CONTEXT 

Over a ten-year period, Heatl, carried out an extensive ethnographic inves- 
t.gnt.on of children earning to use language at home and at schLl in two 
communities only a few miles apart in the southeastern United States. 'Road- 
ville IS a white working-class community of families who have worked in the 
local mills for generations Trackton ' is a black working-class community in 
which there rs a transition between the older generation, who worked as farm 
lalv)urcrs, and the current generation, who work in the mills. 

JUSTIFICATION 

The justification for the research came in the form of legislation mandating 
school desegregation. 1 lie massive shift in students brought about by desee 
regation created a need to investigate how children talk when they come to 
school and what educators should know and do about oral and written 
language. 

RESEARCH QUESTION 

Heath s investigative question, which formed the point of departure for her 
research, was formulated in the following way: What are the effects of home 
..nd community environmenrs on the learning of the language structures and 
functions needed to succeed at school and at work? 
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SUBJECTS 

The subjects were children from two culturally different communities in the 
Carolinas who were learning to use language at home and in their commu- 
nities. Heath also worked with teachers, showing them how a knowledge of 
the ways children learn and use language outside the classroom can enrich 
classroom learning. She also trained them to be their own ethnographers. 

RESEARCH PROCEDURE 

Heath spent several years living among the communities who provided the 
subjects and database for her research. Her role was therefore one of partici- 
pant researcher. As we know from the preceding section, this close involve- 
ment cnchances the internal validity of the research, as the researcher is very 
close to the communities and the phenomena she investigates. Her intimate 
involvement in the community is revealed in the following quote: 

1 spent many hours cooking, chopping wood, gardening, sewing, and minding 
children by the rules of the communities. For example, in the early years of 
interaction in the communities, audio and video recorders were unfamiliar to 
community residents; therefore, I did no taping of any kind then. By the niid-197C)s, 
cassette players were becoming jxjpular gifts, and community memijers used them 
to record music, church services, and sometimes special performances in the 
community. When such recordings became a common community-initiated 
practice, I audiotapcd, but only in accordance with community practices. Often I 
was able to write in a field notebook while minding children, tending fixxJ, or 
watching television with families; otherwise, I wrote fieldnotes as soon as pxjssible 
afterwards when I left the community on an errand or to go to school. In the 
classrooms, 1 often audiotaped; we sometimes videotaped; and both the teachers and 
I took fieldnotes as a matter of course on many days of each year. (Heath 1983: 
8-9) 

Heath thus took care to ensure that the demands of data collecting should 
not alter the normal daily habits of her subjects. In this way she attempted to 
overcome Labov's (1972) 'observer's paradox*. Despite the stringent research 
rules she set herself, she developed an extensive and extremely rich database 
consisting of fieldnotes, recordings and transcriptions, environmental lan- 
guage (such as street signs and notices), children's songs and rhymes, and var- 
ious kinds of written language. 

As I have already observed, ethnographic research is data rich. Studies such 
as the one by Heath, which was carried out over a decade, yield such huge 
quantities of data that only a small fraction can be presented in the written 
refxjrt. This makes it imp)ossible to investigate the internal reliability of the 
research through peer review and other means recommended by LeCompte 
and Goetz. To a large extent, one has to take Heath's interpretations and 
conclusions on trust. 
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TYPE OF DATA AND ANALYSIS 

The data provided by Heath in the published account of her research include 
.n addition to the r.ch descriptive accounts which form the bulk of the text' 

fn rK 1."'^'' ^^""'"'i ''""''^ ^"^ transcriptions. The four figure 

Zm ^"1'""^^"^^ P'-«^h'«l ^«n-epts of print and examples of arithmet" 

L« oZ' I : ^"^'""^^ -•^^'^ «f P--hool cSre^ 

us« of readmg and wntmg. the structure of children's narratives, and the 
like. Table 3 4 illustrates the translation of the social reality of the familiar 
commun,ty domam mto the unfamiliar school domain. 1 have reproduced he 
table because It illustrates the way in which Heath reduced and reformulated 
a huge quantity of descriptive data into an interpretive schema. It also hifih- 
ghts one of the themes of her research - the tension between the contexfu- 
l.sed, personal knowledge of the home, and the depersonalised, decontex- 
suct: is intr "'r'" ''^ "^^^'''^ ^ ^'^^ ^'i-at^onal 

knowTedge! °" ""'"''"''^ d^'om..:ua\is.d 

Only nineteen short texts from Roadville and Trackton are included in the 
written account of the research. The following extract (Heath 1983- 1^6) is 



Martha 
Wendy: 
Martha: 



Wendy: 
Martha 
Wendy: 
Martha: 

Wendy: 
Martha: 
Wendy: 
Martha: 

Wendy: 
Martha: 
Wendy: 
Martha: 
Wendy: 
Martha: 



irl'I'tV "^^^'"/yi dog in the baby book/Who's that' 

Nuf [the dog s name was Snuffy] 

Let s see if we can find another picture of Nuffie. 
// Wendy points to the same picture/ 1 
I pointing to another picture/ Here he is, he's had a bath with daddy. 
I here he is, this is Nuffie. ^ 
All wet. 

Nuffie got daddy all wet too. 
Where's daddy.' 

tt^'iciuf '° "^""^^ '"^ f'"'"'' "l^- he's not in 

Where Nuf 

Nuffie's over to gran 'ma's, he dug under the fence again. 
Bad dog, Nuf, bad dog. 

J':;;; ^ ^ '^-^ ^--^^ p-- ^^^^ 

Nana, nana /pointing to a picture of Mrs. Dee/ 
Yes, that's nana, where 's Nuffie? 
I don't wanna /pushes book away/ 
But look, there's daddy fixin' to give Nuffie a bath 
No. /trying to get down off her bed/ 

a^notlrNX" "^^"^y around the waist/ we'll find 
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Rcsciin l) Dwthods in hwi^iuj^c k\iniiiig 

Sec, look here, who's rh;u with Nuffic? 
/ / Wendy struggles unci begins to cry 1 1 

Heath provides the following interpretive gloss on this interaction: 

Here, Mnrrha, in spire of Wendy's wandering interest and struggles to change first 
Je top,c nnd then the activity, persists in looking for pictures of Snuffy. Once 
V;^cnc y responded to her request for that label, Martha continued it as the topic, 
.> ncJ did .>ot t.kc up Wendy s possible suggestion of Nana (or the finding of pictures 
of otlK-r fx.rs,,ns)as new topic. Thus throughout the conversation, Nuftie is the 

EllTJiri^?! ""^ •''"'^ 

CONC LUSION 

The general conclusion drawn by Heath is that adults help children focus 
their attention on the names and features of particular items or events She 
niso .iscnhcs to her suhjccts the Wicf that teaching children to attend, listen, 
and behave cicve <,ps lx,th language skills and learning skills. She goes on to 
assert that Roadvilie parents believe that preschool age children need to 
develop the abihty to communicate their needs and desires, and that they 
nuist learn to be conmuinicativc partners in a certain mold' (p 127) Shesup- 
ports this assertion with a brief comment from one of the mothers: 

"i hgure it's up to me to give 'im a g.xnl start. I reckon there's just some things I 
know he s gotta learn, you know, what things are, and all that, n you just don't 
happen onto do,,, all that right. Now, you take Danny '„ Bobby, we, Betty 'n me 
we t., k to then, k,ds all the tune, like they was grown-up or something, 'n we try' 
to tell nn Ix.ut things, n UH.ks, 'n we buy those educational toys for 'em. (pp. 

In edncati()na! terms, success and failure are dependent on the child's mas- 
tering the decoiitc.xtualised language of the classroom. 

Contrasting psychometry and ethnography 

111 this section, I shall try to bring together the key issues which have emerged 
over the last three chapters. In Chapter 1 I argued that while some researchers 
(see, for exan,ple, Chaudron 1988) argue for the complentcntarity of quan- 
ft.uive and ciuahtative research, it is possible to discern a gap between the 
two traditions which occasionally widens into n gulf. A major issue here con- 
cerns what one accepts as 'trutli': that is, whether one should admit the sub- 
jective views ami perceptions of the researcher and the researched as valid 
evidence - whether, m fact, one Ixlieves that there can ever be such things as 
obiective lacts whicii are external to the individual. 
I.eC:ompte and Goetz (1982) argue that ethnography differs from expcri- 
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mentation in that data are gathered before the formulation of hypotheses, and 
that the subjective experience of the participants in the research process are 
accepted as valid knowledge. 

Ethnographic research differs from positivistic research, and its contributions to 
scientific progress lie in such differences. These may involve the data gathering that 
necessarily precedes hypothesis formulation and revision or may focus on 
descriptive investigation and analysis. By admitting into the research frame the 
subjective experiences of both participants and investigator, ethnography may 
provide a depth of understanding lacking in other approaches to investigation. 
(LeCompte and Goetz 1982: 32) 

They expand on this contrast, identifying salient distinctions between the 
two traditions in the way in which problems are formulated, in the nature of 
the research goals, and in the application of research results. In formulating 
a research problem, psychometry attempts to identify causal relationships 
amongst variables by extracting these variables from their natural setting, and 
thereby attempting to neutralise or eliminate extraneous variables. Ethnog- 
raphy, in contrast, gives central importance to the context in which the vari- 
ables occur, and emphasises the interplay amongst them. The second distinc- 
tion identified by LeCompte and Goetz concerns the nature of the research 
goals and, in particular, the stage of the research at which theoretical consid- 
erations become salient. In psychometric research, the research questions are 
formulated as hypotheses and the constructs are operationalised in advance 
of the data collection phase. In ethnography, on the other hand, there is an 
attempt to remain as open minded as possible, and there is an interaction 
between questions and data to the extent that it is not uncommon for the 
questions themselves to change in the course of the research (Freeman 1992), 
something which is unacceptable to the psychometrician. Although oversim- 
plifying things somewhat, LeCompte and Goetz have captured this distinc- 
tion rather neatly, suggesting that 'experimental researchers hope to find data 
to match a theory; ethnographers hope to find a theory that explains their 
data' (p. 34). The final area of contrast, that of application of results, relates 
to the issue of external validity, which I have already discussed. Whereas 
experimental research seeks to generalise from samples to populations, such 
statistical generalisation is not possible in ethnography, where there has been 
no random assignment of subjects to experimental and control treatments. 
Rather than seeking generalisability, ethnographers seek validity in terms of 
comparability and transferability. 

Comparability requires that the ethnographer delineates the characteristics of the 
group studied or constructs generated so clearly that they can serve as a basis for 
comparison with other like and unlike groups. . . . Translatability assumes that 
research methods, analytic categories, and characteristics of phenomena and groups 
are identified so explicitly that comparisons can be conducted confidently. Assuring 
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TABLE 3-5 CONTRA STING PSYCHOMETRy AND HTMNOGRAPHY: PKINCPLK. 
I'jychomctry V.th„ogrui>hy 



Hormularing a 
research problem 

Nature of goals 
Application of results 



IdennHcs causal relationships 
among variables by 
extracting from context 
Hypothesis then data 

To generalise from samples to 
populations 



Central importance given 
to cotitext 

Data then hypothesis 

Comparability and 
rranslatability 



TABLE 3.6 CONTRAS TING PSYCHOMHTRY A ND ETHNOGRAPHY: AN EXAMPLE 



Chaudron a,ui Richards (1986) Heath ( 1 983) 



Research What is the effect on L2 
question learners' comprehension of 

lectures of the use of discourse 
markers which indicate (a) 
overall organisation and (b) 
links between sentences within 
the lecture? 

Subjects Two groups of pre-universiry/ 

university ESL students 

Method Experiment 

Type of data Scores on cloze, multiple choice, 
and true/false tests 



Type of Statistical: correlation, 

analysis ANCOVA 



What are the effects of home and 
community environments on the 
learning of those language 
structures and functions which 
are needed to succeed at sch(X)l 
and at work? 

Children belonging to two rural 
working-class communities in 
the United States 

Naturalistic, contextualised data 
collection 

Interviews; fieldnotes; transcripts 
of child-adult child-child 
interactions; environmental 
language 

Interpretive, discursive 



comparability and translatability provides the founditi..n u- l 

are made. (LeCompte and Goetz 1982: J4) "P«" "^^''^ comparisons 

problems, goals, and ^ppi^c::::::^^^:^^::^::^^::^ 

two sers of principles is exemplified by a comparative uZarv rHe r?' 



70 



Conclusion 

In this cliaptcr, 1 have inlrotiiiccil cilim)j;r,ipliy as an alicrn.iiivc rcscanli ira- 
dition to psychomctry. In the first part of the ciiaptcr, I siij;(;(.-stt'd that cili- 
nography was underpinned by very different values and assiiniprions from 
those of psychometry. Two beliefs in particular have guided the evolution of 
this research tradition. These arc the importance of context to luiinan lx-iia\- 
iour, and the centrality of the subjective belief systems of tiiose involvcti in 
research to the processes and outcomes of research. 

Ethnographic research has sometimes been criticised tor its failure to guard 
against threats to reliability and validity, ami in this chapter 1 devoieil st)ine 
time to a discussion of the practical steps which can Ix; taken to guard against 
such threats. While reliability and validity are critical to psyciiometry, some 
researchers have argued that they are not necessarily the appropriate criteria 
against which ethnography should be jutlged. My own view is that all 
research needs to be reliable. Internal validity is important in explanatory 
rather than descriptive research (and is therefore important if one accepts 
Watson-Gegeo and Ulichny's stricture that ethnogra|jliy sliould Ix- explana- 
tory as well as descriptive). External validity is only an issue for researciiers 
wishing to make claims of generality beyond the research sites where their 
data were collected. 

The ethnographic tradition was illustrated by a detailed summary of the 
research by Heath into language learning and use in rural communities in the 
United States. In providing this summary, I have tried to provide some idea 
of the types of questions which ethnographers ask, and tiie sorts of analyses 
in which they engage. 

In this chapter, as in Chapter 2, I have tried to avoid making normative 
statements about the two different research traditions. In principle, the 
research method or methods one employs should be determined by the ques- 
tions which one wishes to investigate, rather than by any predetermined 
adherence to one tradition rather than another. In practice, however, I sii.s- 
p>ect that a preference for one particular tradition determines the types of 
questions one considers worth asking in the first place. Some of these issues 
will be revisited in the next chapter on the use of case studies. 

Questions and tasks 

1. Read the following observations Heath makes on her researcii method- 
ology. In what ways are Heath's comments and orientation consistent with 
her research as exemplified in this passage? 

For my work on children learning language in the two communities, I focused 
primarily on the face-to-face network in which each child learns the ways of acting, 
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Rcsfcirch methods h, Lwgn^jge k-anmig 

c<.n„„uniric.s i„ whi^h S cM ^^^^^^ -ch of rhc 

describe: -socialized as talkers, readers, and writers to 

-3t^SSr """""" 

prcdcsigiKital period of time. What this b<x>k dcvQ ,1^ Jc . ^ , ^ 

~„,ty and classr,H„n life <,vcr nc riy . dec^^^^^^^^^ "T"^ 

1" preschool dnilv cvixTi,-.,,-,.^ ... i ^ .""^ ™'f"<^r '^niid interactions 
2. Select a„ c,h,u.,nphic invcstip.tion from the literature (for example Au 
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Ethnography 

and Jordati 1981; Carrasco 1981; Clegliorn and Genesse 1984). Summarise 
and critique the study by supplying the following information: 

a. Question/hypothesis 

b. Significance/value of the study 

c. Subjects 

d. Procedure 

e. Type of data 

f. Type of analysis 

g. Conclusions 

h. Further research 

i. Critique 

Now, with reference to the questions in Tables 3.2 and 3.3, decide how well 
the researchers have managed to guard against threats to the reliability and 
validity of their research. 

3. Design one of the following: (a) a follow-up investigation to the Heath 
study, or (b) an alternative to the Heath study (this can either be an alternative 

way of investigating the development of language at home and at school, or 
an investigation of some other aspect of language learning and use which 
employs a similar research design). Include the following: research question; 
justification; data collection proccdure(s); typ)e of data; type of analysis. 

4. What do you see as the potential threats to the reliability and validity of 
the study you designed? 

Further reading 

For anyone seriously interested in contextuaiiscd research into language 
learning and use. Heath's (1983) investigation of language and learning is 
essential reading. 

LeCompte and Goetz (1982) provide an articulate and considered analysis 
of threats to the reliability and validity of ethnographic research, as well as 
suggestions on ways in which these threats may be overcome. 



4 Case study 



;'A case study is what you call a case, in case. .„ case you don'r have anything 
J (unidentified student cited in Jaeger 1988) 

- What is a case study? 

- What are the characteristics of single case research' 

- How arethe reliability and validity of case study research tobe established? 



Defining case studies 

tradition in its own richt or merelv i arr^.Ji i a research 
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j their own right. It is teniptinn to .irguc th.n the jccutmil.ition ot c.isc studies .illows 
tliiTii-y liuiKiinn via iciiMlivf hyixuluMs ciillol (roin lln' .Knimiil.iiioii nl siii|'.K' 
• instances. Rut the Kencralisjtions priKiuced in casi.' study aro no loss logiiini.ui- 

, when about the instance, rather than the class from which tlie iiisr.uK'c is drawn 
I (i.e. generalising about the case, rather than from it). (Adehnan ot al. 1976: 140) 

In this extract, the authors argue that the invcstig.ition of .i single iiist.iiKc is 
a legitimate form of inquiry, and that the case study rescarciier need not leel 
bound to report the instance as an exemplar of a class of objects, entities, or 
events. This view, as we shall .see later in the section, is challenged by some 
researchers. 

Having determined what case study is not, Adelmaii et al. go on to suggest 
that it is the study of an 'inst;ince in action'. In otlier vvonls, one selects .m 
instance from the class of objects and phenonien.i one is investigating (tor 
example, 'a second language learner' or 'a science classroom') .iiul investigates 
the way this instance functions in context. From this description, there may 
seem to be little distinguishable difference lx;tween ethnography and case 
study, and, indeed, some researchers apfjear to see the case study as a limited 
type of ethnography (see, for example, the contributions to Bartlett, Kemmis, 
and Gillard 1982). 1 would agree that the case study resemblc-s ethnography 
in its philosophy, methods, and concern for studying phenomena in context. 
However, while the comprehensive case study (for example, of a whole school 
district) may be indistinguishable from ethnography, for most ca.se stuilies, 
the differences are more apparent. In the first place, the case study is generally 
more limited in scope than an ethnography. Another possil)le dilfercnce is in 
the focus of the research. Deriving, as it does, from anthropology, ethnogra- 
phy is essentially concerned with the cultural context and cultural inrerpre- 
tation of the phenomena under investigation (Wolcott 1988). This is not nec- 
essarily true of case studies. Finally, while the case study, like ethnography, 
can utilise qualitative field methods, it can also employ quantitative data and 
statistical methods. 

Smith, cited in Stake (1988), admits that the definition of the case study is 
ambiguous, but states that the term hoioulfd system defines tiie mcthoti lor 
him. " / n >. 

The crux of the definition is having some conception of the unity or totality of a 
system with some kind of outlines or boundaries. For instance, take a child witii 
learning disabilities as the bounded system. You have an individual pupil, in a 
particular circumstance, with a particular problem. Wiiat the researcher lix)ks tor , 
are the systematic connections among the observable behaviors, speculations, 
causes, and treatments. What the .study covers depends parity on what you are 
trying to do. The unity of the system depends partly on what you want to find out. 
(p. 255) 

Adelman et al. suggest that case study research may be initiated in one of 
two ways. In the first of these, an issue or hypothesis is proposed, and an 
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insr.Hicc ilrjwn from rh.it cl;i.ss is selected and studied. In tlie second, ,i case 
is selected and studied in its own right (rather tlian as an exemplar of a class), 
^n' hoth approaches, the case will he a 'bounded system' or 'single instance', 
such as an individual teacher, a classroom, or even a school district. The 
instance niay even be a construct, such as an innovative teaching prograni. 
/While it is relatively straightforward to sec that the linguistic development 
1 of a single individual constitutes a case, there arc occasions (e.g., when inves- 
tigating larger 'lx)unded systems' such as a school district) when the 
researcher is forced to confront the question: Has my study ceased to be a 
case? 

Within the literature, a range of definitions and descriptions is offered, as 
can Ix' seen from the following s;iniplc: 

1 . ' I he study of the speech, writing or language use of one person, cither at 
one point in time or over a period of time, e.g. the language acquisition of 
a child over a period of otic year' (Richards, Piatt, and \Velx:r 36). 

2. '. . . it tries to iliununate a decision or set of decisions; why they were 
taken, how they were iniplcmeiued, and with what result' (Schramm 1971 

. cited in Yin 1984: U). 

^. 'A case study is an empirical inquiry that investigates a contemporary phe- 
nomenon within its real-life context; when the Ixnindaries between phe- 
nomenon anti context are not clearly evident; and in which multiple 

. ^ sources of evidence are used' (Yin 1984: 23). 

4. ' ''I'lie most common type of CS [case study] involves the detailed descrip- 

tion and analysis of an individual subject, from whom observations, inter- 
views, and (family) histories provide the database (Dobson et al. 1981;] 
Shaughiiessy & Zechnieister 1985). CSM [case study methodology J is par- 
ticularly characteristic of sonic areas of psychological research, such as 
clinical psychology, which studies and aims to treat abnormal (e.g., anti- 
stxial) Ix-'haviour.' In principle, though, CSM may involve more than one 
suhjeet (e.g. a series of ClSs, cf. MciscI et al. 1981). It may Ix- based on par- 
ticular gronj;s (e.g. group ilynatuics within a classroom); organisations 
(e.g. a summer intensive language learning program at a university); or 
evenis (e.g. a Japanese language tutorial . . . where one could examine the 
.nnouiit ol tinjc a teacher sjieaks in either Japanese or Knglish for class 
management pur[x)ses)' (Duff 1990: 35). . 

5. 'A longitudinal approach (often called a case study in the SLA field) typi- 
cally involves observing the development of linguistic performance, usu- 
ally the spontaneous speech of one subject, when the speech data arc col- 
lected at peri(xlic intervals over a span of time.... The longitudinal 
approach couUt easily be characterized by at least three of the qualitative 
paraiiigni attribiues: iiatiualisric (use of spontaneous speech), process-ori- 
ented (it takes pi. ice t)ver time) and ungeneralizable (very few subjects)' 
(l arsen-l-icemanand Long 1991: 11-12). 



Crtse study 

6 ' the qualitative case study can be defined as an intensive, holistic 
^ de«:.^^ption and analysis of a single entity, phenomenon, or soc.al untt. / 
£Te studTes are particularistic, descriptive, and heur.st.c and rely heavily ( 
on , nductivereaLingm handling multiple data sources (Mernam 1988. 

16). . ^. , 

Unlike the experimenter, who manipulates variables to determine their , 
causJst t^ificance, or the surveyor, who asks standardised questions of large ; 
™ enS i^^^ samples of individuals, the case study researcher typically . 
S ves th characteristics of an individual unit - a child, a ^-que^ class 

ull or =, rommunitv The purpose of such observation is to probe deeply 
Tndt^ a yseTe inSsI^^^^^^ the multifarious phenomena that constitute 
he life c cle oVthe umt with a view to establ.shmg generalisations about the 
vvid population to which the unit belongs (Cohen.and M^^'^^JJ^^^;^ f 
With reference to the research model from van L.er presented m Chapter 
(I Figure 1 3 . he amorphous nature of the case study .s reflected m the f a ; 
hat there a e s udies in the literature which represent threeof the four seman- 
[iTspaces identified by van Licr (I am not aware of any studies involving for- 

p;o:d:t:researchtothatbasedonsamp..ngands^^^ 
by the writer. 
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TABLE 4.1 THE CASE STUDY: A TYPOLOGY 


Type 


Description 


M 

Neo-ethnographic 


The in-depth investigation of a single case by a 




participant observer 


hvaluativc 


An investigation carried out in order 10 evaluate policy or 






Multi-site 


A Study carried our by several researrhen; nn mnrt- tU-^n 




one site 


Action 


An investigation carried out by a classroom practitioner 




in his or her professional context 



Source: Based on Stenhousc (1983). 



Adelman et ai. (1976) suggest that there are six prmcipal advantages of 
adopting the case study as a method of research. In th^Tst i>lace, in contrast 
with other research methods, it is 'strong jn reality' and therefore likely to 
appeal to practitioners, who will be able to identify with the issues and con- 
cerns raised<Secondly), they claim that one can generalise from a case, either 
about an instance, or from an instance to a class. (We shall consider this par- 
ticular claim in the next section when we look at issuesof reliability and vaiid- 
nty.M third 'strength of the case study is that it can represent a multiplicity 
. otviewpoints, and can offer support to alternative interpretaTbns Properly 
presented, case studies can also provide a database of materials which may be 
reinterpreted by future researchers. Fifthly, the insights yielded by case stud- 
ies can be put to immediate use for a variety of purposes, including staff devel- 
opment, within-institution feedback, formative evaluation, and educational 
policy-making. Finally, case study data are usually more accessible than con- 
ventional research reports, and therefore capable of serving multiple audi- 
ences. It reduces the dependence of the reader upon unstated implicit 
assumptions (whch necessarily underly any type of research) and mak« the 
research process itself accessible. Case studies, therefore, may contribute 
towards the democratisation" of decision-making (and knowledge itself)' 
(Adelman et al. 1976: 149). ^ 

In applied linguistics, the case study has been employedprincipally as a tool 
'I ''aZ ''1^ development of first and second laTiguage learnerrin 
the field of firsr anguage acquisition, thefe'Ha ve been numerous case studies. 
Roger Brown s (1973) longitudinal investigation of the semantic and gram- 
matical development of three children acquiring their first language is prob- 
aNy thesemmal work in the field. Another study which has had considerable 
mfluence .s Halliday's (1975) study of the language development of his own 
child. Studies such as these have played an important part in enhancing the 
status of the case study in applied linguistics. 
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The case study has also played an important part in the field of second lan- 
guage acquisition. Case studies 

' have generated very detailed accounts of the processes nnii/or oiitcoiius of 
language learning for a variety of siihjccis, ranging from young cliililren 111 bilingual 

f home environments, to adolescent immigrants, adult niigr.int workers, aiul 
univcrsify-lcvel foreign language learners. . . . Research questions addresscil in (iSs 
in SLA have included . . . How do children manage to function with two linguisiic ' 
systems at a time when most children are aiteniptiiig to master one? Why do some 

, learners fossilize in their acquisition of a second language (in some or .ill ilomains) 
while others continue to progress? In what ways do the forms and tunctions of 
constructions in a learner's interlanguage (11.) differ? What features characterize the 
prototypical "good language learner"? How ilo learners react to aiul/or Ix-nclii 
from different methods of instruction? Is there a criiical period for SI, A? (Duff 

I 1990:34) 

One widely cited study in the second language area is Sclunnami (1978), 
who investigated the hypothesis that second language dcve!i>|jnieiir will he 
governed by the extent to which the learner identities with and wishes to 
acculturate with the target language community. Schumann carried out a 
longitudinal case study of Alberto, a 33-year-old Costa Rican, who niadf little 
progress in learning English despite intensive instruction. Schiiniaiui con- 
cluded that Alberto's lack of linguistic development could be attributed to his 
social and psychological distance from the target culture, and the fact that 
his limited English was sufficient to enable him to fulfil his coniniiinicative 
needs. A more limited and focused case study is Sato's (1985) investigation of 
the phonological development of an adolescent Vietnamese learner of Engli.sli 
as a second language. Later in this chapter we shall look in some detail at a 
case study by Schmidt (1983). which took Schumann's research as its point of 
departure. 

In this section. I have reviewed a range of definitions and descriptions of 
the case study as a method of research. We have seen that a case is a single 
instance of a class of objects or entities, and a case stutiy is the investigation 
of that single instance in the context in which it occurs. The contextiialised 
nature of the case study, along with the types of data collection methods 
which arF typically used, make it similar in some respects to ethnography, 
although I have tried to indicate the ways in which the two types of research 
may differ. A key issue for this type of research is the extent to which the 
insights generated by the study can be applied to other cases. This and other 
issues are taken up in the next section. 

Reliability and validity of case study research 

As case studies are concerned with the documentation and analysis of a single 
instance, many of the issues we looked at in the preceding section, and in 
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C.hnprcr 3, w,ll be pcrtmcnt. There are two points of view on the issue of 
vn .d.ty On the one hand, there are the researchers who feel that, wh. i inter 
M vnM.ty .s .mportnnt, external validity, may be irrelevant. 6n the^r 
hn„d other researchers take a different view, arguing that tests of validity 
should be as stringently applied to the case study as to any othe lype of 
rescarcli. Stake (1988) represents the first view: 

The prmcipal difference between case studies and other research studies is that the 
ocus of artenfon ,s the case, not the whole population of cases. In most other 
studies researchers search for an undcrsrand.ng that ignores the uniqueness of 
...drvdual cases and generalizes beyond particubr instances. They seS for what 
.s common, pervasive, and lawful. In the case study, there may or'may not an 

unKuc .nrcresr n, rhc gcneralizable. For the time being, the iarch is for In 
1 9«r^^^^^^ ' P-Tt.cular case, in its idiosyncrasy, in its complexity. (Stake 

Yin (1984). however, klieves that rehability and validity are just as impor- 
nnt for case study research as for any other type of research. He suggests \hzt 
four critical tests confront the case study researcher. These are: 

- construct validity (establishing correct operational measures for the con- 
cepts Ix-mg studied) 

- internal validity (establishing a causal relationship, whereby certain con- 
ditions are shown to lead to other conditions, as distinguished from spu- 
rious relationships) . ^ 

- external validity (establishing the domain or population to which a study's 
hiidings can l>c generalised) • ' 

~ r'eluw''' ^'^''""'"''""'"S ^'^'^ ""^y ^ replicated with^imilar ' 

Yin nrgues that construct validity is especially problematic in case study 
research. Ih.s is due to the frequent failure of case study researchers to 
develop a sufficiently operational set of measures and because subjective' 
ludgmcnrs are used to collect the data. 

In relation to the internal validity of case study research, Yin claims that 
h s ,s a concern only for causal or explanatory studies, where an investigator 

s trying to deternn ne whether even t X led to event y. If the investigator incor- 
ectly concludes that there is a causal relationship between .v and y without 
knowing that some third factor - z - may actually have caused y, the research 
design has failed to deal with some threat to internal validity' (1984- 38) 
Another probicn, relating to internal validity is the frequent necessity fo; case 
study researchers to make inferences (researchers are required to make infer- 
ences every time they deal with an event which cannot be directly observed) 
I hus, an investigator will 'infer' that a particular event resulted from some 
earlier occurrence, based on interview and documentary evidence collected 
. s parr of the case study. Other researchers argue that internal validity Tof 
concern in all types of research, because it deals with the question of whethe 
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investigators are really observing what they think they are observing. For 
Cuba and Lincoln (1981), internal validity takes precedence over external 
validity because without internal validity results are meaningless, and 'there 
is no point in asking whether meaningless information has any general appli- 
cability' (p. 115). 

The third test identified by Yin relates to external validity, that is, knowing 
whether a study's findings can be generalised from the particularities of the 
immediate case study to a more general context or situation. This problem 
has been a major stumbling block for the case study researcher because of the 
obvious difficulty of arguing from the single instance to the general. Yin deals 
with this potential threat by arguing that it involves a false analogy with sur- 
vey research. 

Such critics are implicitly contrasting the situation to survey research, where a 
'sample' (if selected correctly) readily generalizes to a larger universe. This analogy 
to samples and universes is incorrect when dealing with case studies. This is because 
survey research relies on statistical generalization, whereas case studies (as with 
experiments) rely on iz«a/y/;ca/ generalization. (1984: 39) 

I must say I find Yin's line of argument on the major threat to case study 
research (that is, in relation to external validity) a little obscure. As wc have 
seen, a major barrier to doing case studies concetns the extent to which^a 
particular finding can be generalised beyond the case undej^j^nvestiga tion. The 
issue of making generalisations from instances is complex, and thcrc..is^ 
branch of higher mathematics devoted to the subject.. (See also the discussion 
on statistical inference in Chapter 2.) The thrust of much scientific work is 
towards 'predictive generalisation' through a process of theory construction 
and testing (usually through some form of observation). Yin has dealt with 
this potential threat by claiming that case studies seek to make 'analytic' 
rather than 'statistical' generalisations. This brings to mind Popjjer's work on 
falsifiability (for a discussion on this, see Chapter 1). 

It seems to me that Yin is arguing that case studies are appropriate as a tool 
for falsifying a particular hypothesis or claim on the grounds that a single 
disconfirming instance is sufficient to refute a given hypothesis or claim 
(providing, of course, that it is formulated in a way which enables it to be 
falsified - this is by no means true of all hypotheses). If this is an accurate 
reading for Yin, it would seem to cast the case study into a rather negative 
role, one reminiscent of the 'weak' view of ethnography which we looked at 
in the preceding chapter. 



Single case research 

Single case research represents a special type of case 3ludy, which is why I have 
chcjsen to deal with it in a separate section. It shares some charaacristics with 
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No. of 

observations 




Baseline (A) Intervention (B) Base (A) Intervention (B) 



Figure4.1 The four stages in an ABAB single case research design . 
' !l!."'u 'I'^'-ibed so far, and also wirh experimental 

resulr. The difference between experimental research as described 

2 and single case research is that experimental studies typicaliy^n "olve com 

panng two or more groups, while single case research, like the case stX 

4.1, we can see that rhe disruptive behaviour^clin./^ I ' '^"'^ 
phase. (At this point the '°"'^''^^''""^"'''"g f^e mtervention 

Ldiiy thatitisrt:;^:r:^^^^^^^^^^^ - — - 

betharthechildshomecircumstSeshave^^^^^^^^^ 

be growmg up. If the researcher were ro ascrifi ^'"'1 "^^^ '""'P')' 
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would say that the internal validity of the study was ;u risk.) in tiic next 
phase, the treatment is withdrawn, and the conditions wliicli t xisti d during 
phase A are restt)red. The researcher now wants to know wlKther the Ix'hav- 
iour will remain at the (in our example) lower level predicteil by the interven- 
tion phase. In the final phase, the intervention is restoreil, and the individual's 
behaviour is observed once more. 

The logic of the ARAB design and its variations eonsists of ni.iking .iiul testing 
predictions about performance under different condition!;. I'ssenti.illy, (.l.il.i in the 
separate phases provide information about present perforiiiaiue, predict the 
probable level of future performance, and test the extent to which predictions ot 
performance from previous pluses were accurate. By re|xMtedly .ilteriug 
experimental conditions in the design, there are several diflerent o|j|K)riinnties to 
compare phases and to test whether performance is altered by the intervention. If 
behavior changes when the intervention is intrixluced, reverts to or near baseline 
levels after the intervention is withdrawn, and again improves when treatuieiit is 
reinstated, the pattern of results suggests rather strongly that the intervention was 
res[>onsible for change. (Kazdin 1982: 113) 

In the example I used to describe and exemplify the single ca.se procedure, 
only a single subject was involved. However, when a re.searcher carries mit an 
investigation in order to make generalisatioas, rather than for theiaptutic or 
clinical reasons, it will be necessary to collect data from more than one sub- 
ject. Barlow and Hersen (1984) point out five difficulties, problems, and objec- 
tions associated with the use of a single case design for such research purposes. 
These are ethical objections, practical problems associated with identifying 
large numbers of subjects (particularly for clinical research), averaging of 
results over the group, generality of findings, and iniersubject variability. 
These problems are set out in Table 4.2. 

The case study: an example 

In this section, we shall examine a case study of an adult learner of KSl,. The 
study is from Schmidt (1983). 

RESEARCH AREA 

Schmidt set out to explore the relationships l')etween S()ciai and interactional 
variables on the acquisition of communicative competence. 

JUSTIFICATION 

At the beginning of his paper, he points out that most current research is 
biased towards the acquisition of morphology and syntax, to the virtual 
exclusion of semantic and pragmatic aspects of second language ilevelop- 
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Priihlnn 



I'.tlnctI olijccrions 



''nitticn; prohlciDs 



Avcraf^iiiKofri'siilis 



(.'cncraliiy i)f liruliiigs 



Inrcrsulijcci v.in.iliilif y 



CoitlHlClll 



I" ^■'''y"' 7^cnrch, ,t mny Ix. a»,s,dcrcd unethical 
t" w,rh;,olcl trenrmcnt fn.n, subjects during 

rnlfr.""' grounds rhnr ir 

.-.ssumcs the rrenrmenr wcrks wl,cn, prcsun,.ibly 

T)ic collection of Inrge numbers (,f subjects with 

s.n,.lnr chnrncteristics is often e«ren,c y dtfficdt 

pnrt,cnl.,rly,„cnses where some form of 
[Wthology is involved. 

' Ik- .ivcr;,gi„g „f results cnn (x; a problcn, if the 
researcher is suuult.neously trying ,„ derive 
Snicrahsable results .,nd obtnn, d.to on spechc 
sub,ect,sf„r,l,,,g„„sisorreMudin,ion The 
•1 vcnigmg of results okcures the performance of 
.nchvKlunl subiects and. as the over-performattce 
o( son,e s,,l,,ects can k- cancelled out by the 

■"Hicr-performance of others, g.vesa non- 
sigiiihcant result. 

This problem ,s related to averapinR. Because 
resu ts frou, group studies do not reflect changes 
'-"ul'viJuas,,n.sdif]f,cnlt for. he researcher 
who wants the data for clinical pur,x,ses, to ' 
cIcternuMct,, what extent any given subject is 
snnilarrofhosein the group who .showed 
unprovetnent. 

As vve have sa-n, son.e subjects may unprove 
while others deteriorate or remain stable. 



'"«••>'■ I" Ins li.cunirc review ho refers tc. the work of Hitch f I97«l , u 
whomaiiunm t lar synncnr .rr.--h„. , i , "^™f^""^78J and others 

-ntret,, tll.s centra ted wi i ; :e7, T ^'"^ ^'^'■^ 
"■res, and then chains' t^^ r ,ct r 7" ""^ '"^^ ^''"^'^ 
r-icwed arc studies rest n^ d, Z ^ 7 ^''^^ 



tion, and that these may determine the amount of acquisition. Schmidt con- 
cludes from his review that 'there is an assumption that if communicative 
needs were greater, and social distance less, much greater control of the gram- 
matical structures of the target language could have been acquired without 
formal instruction' (p. 139). 

BACKGROUND 

In this case study, Schmidt sought evidence for the acculturation model by 
carrying out a case study over a three-year period of a learner with low social 
and psychological distance from the target culture who was acquiring the 
language naturalistically, that is, without formal instruction. The subject, 
Wes, was a native speaker of Japanese whose positive attitudes to the target 
culture were predicted to facilitate second language acquisition. Schmidt 
made this assessment by examining factors such as attitude, culture shock, 
and empathy, although he points out that such psychological factors arc 
extremely difficult to operationalise or evaluate, and that they are all subjec- 
tive, some highly so. For example, on personality variables, Schmidt says, 'All 
observers agree that Wes is an extremely extroverted and socially outgoing 
person, with high self-esteem and self-confidence, low anxiety and inhibition. 
He is highly perceptive of the feelings and thoughts of others, intuitive, rather 
impulsive, and not at all afraid of making mistakes or appearing foolish in his 
use of English' (p. 142). While such highly subjective observations cast doubt 
on the internal validity of the study, it is difficult to see hbw they might have 
been obtained in any other way. 

The theoretical construct for the study is provided by Canale's (1981) four- 
component model of communicative competence. This model specifies gram- 
matical, sociolinguisric, discourse, and strategic competence as the basic ele- 
ments constituting a user's overall competence in any given language. Gram- 
matical competence is glossed as the elements and* rules of the target 
language, including word formation, sentence structure, semantics, pronun- 
ciation, and spelling. Because of Wes's limited competence, Schmidt only 
looks at pronunciation and grammar. Someone who is sociolinguistically 
competent in a language is able to produce and comprehend utterances which 
are appropriate to the context in which they are used. This appropriateness 
can relate to either meaning or form. While grammatical and sociolinguisric 
competence relate to language at the sentence level, discourse competence 
refers to mastery of the ways in which forms and meanings combine to 
achieve unified spoken or written texts. As Wes was unable to write, it was 
only possible to study the development of his spnaken discourse competence. 
The final component of Canale's model, strategic competence, refers to the 
verbal and nonverbal strategies which are called into play in order to repair 
conversational breakdowns, and otherwise keep an interaction going. 
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TYPE OF DATA AND ANALYSIS 

In common with many case studies, Schmidt draws on several data sources, 
including taped monologues and dialogues, fieidnotes, tables of niorphosyn- 
tactic items, and interviews. Grammatical competence was investigated by 
studying the development of pronunciation and the mastery of eleven gram- 
matical morphemes. Schmidt claims that Wes's pronunciation is 'better than 
that of the average Japanese student I have encountered', although no evi- 
dence is provided to support this claim, and the reader must therefore take it 
on trust. In contrast, data are provided to support Schmidt's claim that there 
was little progress in the acquisition of nine grammatical morphemes. Using 
the criterion of 90% accurate suppliance in obligatory contexts, Schmidt 
claims that over the period of the study (almost a year and a halO no mor- 
phemes moved from unacquired to acquired status. Three morphemes - cop- 
ula be, progressive -ing, and auxiliary be - seemed, on the surface, to have 
been acquired at the beginning of the study, being supplied in most cases when 
they were required. However, Schmidt questions whether the progressive and 
the auxiliary have really been acquired. In order to probe Wes's metalinguis- 
tic knowledge, Schmidt asks him the difference between 'paint' and 'painting' 
to which Wes replies: 

Wes: Well if 1 go to exhibition, I saw 'paint', but 'I'm start painting' means I do it, 
not finish. 

RS: Yeah, OK, sort of, so what's the difference between 'chink' and 'thinking? 
Wes: 'I'm think' means now. 'I'm thinking' means later. 

(p. 147) 

If we accept the validity of introspection, then there would seem to be pretty 
clear evidence here that Wes has not sorted out the distinction between the 
two verb forms, at least on a metalinguistic level. 

Wes's socioiinguistic competence was evaluated through a discursive anal- 
ysis of his utterances as recorded in Schmidt's fieidnotes. Schmidt focuses on 
Wes's use of directives (that is, getting others to do things at his behest) and 
claims that in the early stages Wes was reliant on a limited number of for- 
mulaic utterances, but that there is evidence of development over time. He 
argues that Wes's improvement in the area of socioiinguistic competence 
reflects his high motivation to engage in interaction, and his desire toaccul- 
turate with the target society. 

Discourse competence, that is, the ability to produce coherent texts, is 
Wes's greatest strength and the area where the greatest improvement is evi- 
denced over the duration of the study. The database for this aspect of devel- 
opment is a series of taped conversations and monologues. The type of data 
collected by Schmidt, and the interpretive analysis to which he subjects it, is 
exemplified in the following extract and commentary. Schmidt claims that 
the extract demonstrates Wes's skill at conversational small talk. In the 
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extract, he is chatting with a married couple whom he has only ,ust nu t at 
hotel garden brunch. 



I would like e,.s l.n.l.t (to wa.ucss) / tlu.'s ,..e s,.eculi.y (.o We.) 

Waitress: how atx^iit you? 
Wes: here eggs benedict is gwxl? 
yeah 

it's the speciality 

yeah? / OK / 1 have it (waitress leaves) 



M: 
G: 

Wes: 
M: 
Wes: 
M: 
Wes: 
G: 
Wes: 
M: 
G: 

Wes: 
G: 



you never ate before? 
no late before /but not this hotel 

I difcren, /»/■!.» rc.u..'..» » l«l'>"e >" I'""" 

No ; noi exacily (sthmiJi 198.1. IS9-160) 



Schmidt comments; 

The good-natured, teasing type of humor of ^^J^:;':^^^^.... 
inev'ably less obvi.>usfr<jnja^t.ns^ !o wl.u 

tone of voice) is typ.ca! of Wes s Y*"?''T'" , ' .ly Wes is not a i>assive 

people say and picking -P'" 
conversationalist but nommates topics treque y ^^^^^^^^ _^^^y 

nominates are aim..t always relevant ;> , ...pic (or 

instances of conversation comu^t > ^ ,„j 
commented on a top.c already on 'J'^ "'^ ' J^^^^^ j^^j „nf„l„,„,able 
understood what the P-.-^^^ff/Xa k" ^^^^^ "'^^'""'^ , 

' 

The final com^^nt of c_ 
strateg.ccompetence:the b hty to us^^^^^^ commumcation. In exam.nm 
strategies ro comfKnsa.^^^^^^^ draws principally on conversatio 

this aspect of W^J^^^^^ '^^at given Wes's limited grammat.cai con 
tapes and fi^''^""'"' "^^t^kdownf are not uncommon, but that Wes 

L« study, short »Xr?„trg exat^^^to^^^^^ *a, 



Rcsccnch methods i„ hmguagc learning 

«ivingn,,pcdficc,a„,ple„fwh,!;hc "i yo"' We', and »l» by 

"«./ir;';:,:'Lt7,r'"'' -^'"^ ' „„„ , 

NS: you nic.iM nftcr I dic^ 

"""^ '""^;t ^ ^ ^^^^ 

NS: security I supp<«c / n.L " . ^ " ^ '^•''-'^ 

tlu- Kcint Z ''"""'^ / -^'"^''"gh that looms large at 

(p. 165) 

(.'ONCLDSIONS 

Hiivin}^ provided selective extracts fron, l,ic 

".entcd «,n these, Schn.dt dnms .s co .I^'h "(''''^'^ "^"'^ 

"ot <,nc cnsidcrs Wes to he , nooLZ T ^'''^^ or 

one s de,.„,t.o„s. He cite; ^^^^^^f ^cpct,d ot. 

sociol.nRu.sts'Wicvcthnt WesisTs,m.ri I . ""^^^ '''^^ '"^^^a' 

'Kcnernliy consider hin/a d^s stc ' b 'L L""^^ 

l^ypcnhesis thnt there ,,s n en I e htfonsilr ''I' ^^'^"^ ^^e 

^'f-" nnd Krnn„nnticnl devclopmem ^ ^^^^^ the degree of accultur- 

r — ^ Schtnidt 

An,ct,.,n for the cnse st d 1^^^^^ 

Hypothesis. Having fo,„d a sitiKKehlv V"*-' " f^'^'^^^'^ "^^^''''''^d 

pcrmd of ti,nc, Sch.nidt is able to c J nto n ^ T''"' ' ^'^"'''^""t 
cs,.s: '1 he idea that if aftcxt, e f^ctor^^^^ """""""u ^hcacculturatioti hypoth- 
fi"K-tio,KU.t<.,uuicnlly,X/£ 

""iu-r ..nphcari„„s of the study mcln 1 IT ■ . '^^''^ "''^ ""mcrous 

of . second langt,agc invc> mm " e '"•^'^''^ ^''"^^ ^''^ developti^cnt 

th.s sh:;t,iX re.ic:::r ^ilss:';:^ "^^^^^-^^-^^^ 

Conc/usion 
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studies arc particularly suited to the types of action-oriented research projects 
described in Chapter 1 where the purpose is, in the first instance, to help prac- 
titioners enhance their understanding of, and solve problems related to, their 
own professional workplace, and where the problem of external validity is 
less significant than in other types of research. In particular, one can learn a 
great deal about one's own students in general through a detailed study of 
one particular student, in the same way as insights into language classrooms 
in general can be derived from the intensive analysis of a single classroom. 

Questions and tasks 

1. How convincing do you find the following discussion on the external 
validity of case study research? 

Many people criticise case study research because there is t(X) little indication of the 
degree to which the case is representative of other cases. Usually it is left to the 
reader tt) decide. Of course it is easy to argue that a sample of 'size one' is never 
typical of anything, except itself. 

For some research purpt)scs, it will be essential that the 'cases examined' be 
representative of some population of cases. Presumably, a case could be so unique 
that it might be unwise to consider any Hnding as true of other cases. However . . . 
the unique case helps us understand the more typical cases. 

Whether or not a case should be representative of other cases depends on the 
purposes of the research. It would be presumptuous to dismiss all findings as invalid 
because the case was not demonstrably representative. Some findings - for the 
purposes some readers have - do not depend on the notion of generalizing to a 
population of cases. 

... A case study is valid to the reader to whom it gives an accurate and useful 
representation of the bounded system. Accuracy of observing and reporting is not a 
matter of everyone seeing and reporting the same thing. Observers have different 

vantage points Readers have different uses for research reports. One reader 

expects an exact facsimile of the 'real thing'. Another reader is attending to a new 
type of problem that had not previously been apparent. J hc validity of the re/jort is 
different for each, according to the meaning the reader gives to it. (Stake 1988: 261- 
26.?) 

Can you think of any studies in which representativeness might lie 
unimpt>rtant? 

What is your reaction to the notion that validity is in the eye of the beholder.' 

2. Design a study using the single case research design described in the sec- 
tion 'Single Case Research' in this chapter. 

3. Make n list of the threats to the internal/external reliabih'ty/validity of 
the Schmidt study. How would you guard against these threats in a replica- 
tion of the study? 

4. Select a case study from the literature (for example, Schmidt 1983; Schu- 
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mann 1978). Summarise and critique the study by supplying the following 
information: 

a. Question/hypothesis 

b. Significance/ value of the study 

c. Subjects . 

d. Procedure 

e. Type of data 

f. Type of analysis 

g. Conclusions 

h. Further research 

i. Critique 

5. Carry out a detailed case study of one particular student over a period of 
time (if possible, over a semester or term). Keep fieldnotes, interviews, tran- 
scriptions of interactions between the student and yourself/others, samples 
of written work, and, if possible, get the student to keep a journal or diary. 
Some of the questions to address might include the following: 

- Is there any evidence of language development over the course of 
investigation.' 

- What affective factors seem to facilitate or impede language development? 

- How much target language use does the learner engage in inside/outside 
the classroom? 

- What factors in the learner's home environment seem to have an effect on 
the learner's language development and attitude towards language 
learning? 

Further reading 

One of the best books on case study research in education is Merriam (1988), 
who manages to provide a practical guide while at the same time dealing with 
theoretical and conceptual concerns surrounding the reliability and validity 
of case studies. Another book length treatment of the case study as a research 
method is Yin (1984), who is somewhat more theoretical. However, some of 
his arguments and lines of reasoning are rather hard to follow. (In the body 
of the chapter, I dealt with Yin's rather idiosyncratic approach to the threats 
to external validity posed by case studies.) 

The study by Schmidt (1983) is worth looking at as an example of a detailed 
investigation of the language development of a single learner. Finally, some 
interesting case studies into language learning and deafness can be found in 
Strong (1988). 
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5 Classroom observation and research 



commonplace. Walker, ciied in Sicnliou!.c l'>75: 150) 

. ^m. are soecifically constituted to bring alxnit learning, .t 
As language classr(X>ms are specincaiiy ^^^^^^ ^ 

is not unreasonable to collec data ^'bout what y ^^^^ 
adding to our knowk-dgeo langvu.j^ek rm^ ^^^^^^^^^^ 

shall review the substantial, ^"f^ « ;;^;,X-ping in nm^^ 
research on and about language classroom ''ecpint, 
ho""'s (1975) stricture that there .no te.hng 

Thischapterisrather ta^^^^^ ^^^^^^^^ 
on a research context the ^'^'f .^^^^ froni formal experime.us 

Most of the methods dealt with in ' '^'^ could W u,i- 

through case studies to '"^"-'J^"^:,",!^^^^^^^^^^ the classroom. Tlu 

-Whatistheplaceoftheform^exj^i^n^^^ 

_Sr£2interactionana,ysis^t^;;cla^- 
- What are the strengths anj weaknesses ol currii 

Methods of classroom observation and research 

,„ thU section. . sha« '~""*J,-:renl xSe"^^ 
and interaction analysis. 
The formal experiment 

While(o,malexpetin,entsa,,a,u,s.-expen.™^^^^^^^^^^ 



research). I.ntcr i„ the chapter I s n /' ^^o'l^cting dnta for 

^vlm-l, ,r vv.,,s found tIku „f fif;v '''' ' '^"'■^''•■>' «^ ^'^<^ent research in 

r<,ok p,.ce in « H X:r^'"n^^'/''^ 

f,..., cxpe;- ^ ^ ^"^^ :r — car- 

vcnc herwccn rl,c independent and don ' '^^'^^^ "'-^V .nter- 

rcsults unintcrpretablc'or, at ) e vol " sr"' T'"'"" "^"^ ^'^"^ -"^^ the 
«'Mrd a,a,„st threats t.> the intern v' I J ' ^"'''^'"'^'y '^'ftcult to 

;;;s;^..;^o-ea^:ith:^';:~^^^ 

c^u,ve cla,n,,s „f d.Yferent n.eth.W w T' ^ ''"^ ^^•^'"''•f'^ the 

^''f^^-rent groups and providing ch^ti il ' ' ■^'"''^"'"8 «"d-us to two 
end of the researc/, per.od. al'stnden 1 ? ^^""P''- the 

co,„pet,n, .methods .s the Jrc ZZ '"^""'"^ ^''<=^^ the 

ts^h :sr :^ ^'v--'--^ -.h 

trjui,tn,„anan,uageH.strncric.r,:StS^^^^ 

1969 Suh,ects .„ this stndy ^-e e Cnmn^ ' t'""'' ^^'^^^ 
school learners of fVcnch and Term '"fernicchare level hieh 

"-rncri.;nal conditions: ' "di^^ ^ r'",,: '^^'^^ f to one ol utc 
n»r\ Thcscale,.ftlK- study rs ' "''"^""^' ' •''"^ '^"diohngual plus 

s.^,msi„h.ty-eighth.gh;:a:r;^^ 

t.incc of collecting data on what actn.l ^^re aware of the impor- 

'"t.. the study systen,atic cl ' n ''"^ ^'■^ssroon.s, and bu" It 

7«^'-^PP<.intn,ent <,f tl^ XT""' '^"f'"^'^^- ^" ^'^ - P^- 

cl.Kled that, after two years of -Pensive study co 

c-"a.slx.,ween the ,„ethods u, theare s "'t ' 

^''''.oni^htiu-Vaclit-on iw^ 
••"'■"'Pt'-'J^ t<. :.c-co,„,t for the ItZ ^ ' '"'^^'''^ ^^'-^^'"8 
; '-^-'-n observation -n.ponent Xtelrch'o;"'-^' "'^'^'-'^ ^^e 

H'onis at randon, and noted, o„ f ve 'ohu se , '''"^ ^''^'^^^ '^'^ <-'-^«- 
cTs conformed to the „,cth„J r| 1! ^ ''^'^ "^^nt to which tcich 

"-ly. d.l^.ent rat.ng .^J^l^^^ ^ ^ -^h.ng. U^t 

-led con,pa tisons in,poss,hle, as e ^o'"' '^"^ ^f'- '"-^de 

""tc the use of audiohno,,,, tcc'hninn.? -P'""''''''" ^''^ observer to 

versa. Cl,ri< suggests thauher t S'of?;;'""'"' - 
.."d ti,at this eo„tan,inated the re ul .'^ """'^T ''^'^ "'^'^•^"'^'^ '"eth- 
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TABLE 5.1 EMPIRICAL RESEARCH OF Li CLASSROOMS 



Category 



Goal 



Principal research methods 



I. Classroom process The understanding of how the The detailed, ethnographic 



research 

2. The study of 
classroom 
interaction and 
L2 acquisition 



3. The study of 
instruction and 
L2 acquisition 



'social events' of the language 
classroom are enacted 

To test a number of hypotheses 
relating to how interaction in 
the classroom contributes to 
L2 acquisition and to explore 
which types of interaction best 
facilitate acquisition 

To discover whether formal 
instruction results in the 
acquisition of new L2 
knowledge and the constraints 
that govern whether formal 
instruction is successful 



observation of classroom 
behaviours 

Controlled experimental 
studies; ethnographic 
studies of interaction 



Linguistic comparisons of 
L2 acquisition by 
classroom and naturalistic 
learners; the experimental 
studies of the effects of 
formal instruction 



Source: Based on Ellis ( 1 990b). 



In his overview of observation in language research, Allwright (1988: 10) 
suggests that perhaps the wrong question was being asked: 

[This research was conducted] on the assumption that it made sense to ask 'Which 
is the best method for modern language teaching.'*, and that presumably on the 
additional assumption that once the answer was determined it would then make 
sense to simply prescribe the 'winning' method for general adoption. 

Following the failure of studies such as the Pennsylvania Project, both these 
assumptions were questioned. According to Allwright, it was felt that the 
inconclusivcnessof the results indicated that perhaps the method variable was 
less important than had been thought and that it was therefore meaningless 
to ask which njcthod was superior (sec also Swaffar, Arens, and Morgan 
1982). The Pennsylvania Project also showed that prescribing a method and 
expecting it to be faithfully followed was simplistic and naive. 

The time was ripe, then, for an alternative approach to classroom language learning 

research, an approach that would no longer see the language teaching world in 
terms of major rival 'methods", and one that would be more respectful of the 
complexities of the language teacher's task. (Allwright 1988: 10) 

Similarly, Ellis (1990b) identifies dissatisfaction with global method compar- 
isons as the impetus for a closer focus on the classroom itself as the locus of 
research. He suggests that there have emerged three different categories of 
empirical research, each with its own goal and principal research methods. 
These are reproduced in Table 5.1. 
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-c.c'..c/. language leanmi> 

'•^h a relationship between the dlZTlTj 1^"'^^^"'^'"' to estab- 
and .ndependenr variables such asrnov " ^ f ''"^"^^'^ P^^'«--'ncy 
nor to say chat the task .s ^mpossibr )a er .^f 'T''- 
some interesting Canadian res^h wh ch h '^'P'" ^^'^l' '^^ at 
act.onandlearn.ngoutcon,es S^weveJ het'ri"-'^^^^^ to Jink classroom 
panson studies of the sixties aJerte7rreich-r"T'' '''^ 
prec.se records of what actu^ TcS n ''1" '''^ '^"^''^^ ^"d 
'nvest.gation. '"""'^ 'n the classrooms under 

Stimulated recall 

te^ch.ng and learning which wodd bL'^Sirrn I"''^u P^'^-^^ 

a part.cularly useful technique in coIlT ''>' "f'^^'" '"'^ans. Jt 

Teachers and students as well as hT .etar^.^^^^ --arch because it enables 
pretanons of what is going on in theT P'^"^"' ^^rious inter- 

tobe /inked expl.citly'to "n ^heT"""' '^"^ '"terpretadon, 

recent invest.gations of teacW dea^o!""" C 
"•que. In the firs, of these, WcSs (S""" '"^ ^"^P'^^^d 'his tecT 
-ght ESL teachers. He used^ e I 'a X""'''" '^'"'"'"^ 
nterv.ews. ethnographic observrtL„ " ver tt " '^'^nographic' 
third method i. described in the folio Jng wLy;' ^"^ ^^'-"'^'-d recall. ^This 
[ Stimu lated recalf ? elicited teacher. ' 

decisions made a.d actions takenTn hTcZ "P"''"' -"="dered. 
subsequenriy viewed and cornel J JJ^^ZT' ' ' / '^^^ ^''^-taM and 

composite videotape as a vo.ce-over) the r-^K ^ T""'"' ^"P'"^^d on a 
lesson, the students or about what s/he w^s ' ,7 '''"f ' ^°-'"-tary about the 
The co,r,pos.te videotape containing theT^/ V*!'''' 'inspired 

'nteraa.on, .nciuding the following ^'^'^toom 
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V. ./...-J/ 1 ,, 

2. In terms of procedures for course piaiiniiig, the most sur|jrisiiig liiu 
was the tenrativeness of teachers' advance planning - 'lessons v 
sketched out only in very vague terms and detailed planning occiirroi 
most a couple of lessons in advance even by the most org.mi/al ot 
teachers' (p. 116). 

3. Basedbn an analysis of the teacher interviews, WckhIs concluded that c 
teacher's course was internally coherent. He cites as an example one o( 
teachers whose decision-.naking was driven by her tlesire to tlevelop 
independence of the students as learners. 

4. The final major point to emerge was the fact that different teachers I 
quite different approaches, criteria for success, and so on, and thnt Jit 
ent teachers could take identical materials and use tluin in class in v 
different ways. 

In the second investigation, 1 looked at a number of different aspects of 1 
guagc pedagogy, including teachers' decision-ma ki.ig (Nim.in I*-)')!,!). 1 
experienced (4 to 15 years' experience) and four i.iexperienced (less than < 
year's experience) teachers took parr in the study. Theit lessons w 
observed, audiotaf)ed, and transcribed. Following the lesson, the tend 
were interviewed and invited to comment on any aspect of the lesst)ii ti 
liked. After the transcriptions were completed, they were returned to 
teachers for comment. 

The following extract from the study is taken from a listening lesson. I 
lesson was based on a tape recording of a scries of authentic interviews. In 
classroom extract, the teacher focuses on the questions used by the im 
viewer and requires the students to use 'proper' or 'full' question forms. 

Classroom extract: 

T: OK, bcfi)re we start, I want you to ask the proper i|iics(i<)iis. So please lioii'l ji 
look and copy. What you have to do is . . . OK, Keith, hirst question. Whai 's 
the first question.' What . . . (hours) hours does he work.' Interests. What's the 
question? 

S: What are his interests. 

T: What... 

Ss: are 

T: .... his interests. Again. 
Ss: What are his interests.' 

T: What are his interests. Next question. What's the question? What . . .? 

S: What kind of books . . . 

T: What kind of books - does he read? - Smokes? 

Ss: Do you smoke? 

T: Keith. Does he smoke 

Ss: Does he smoke? 

T: Next question. 

S: Does he drink? 

T: Does he drink? -Church? 



, '^•^''•^«"ve got five 

stimulated recall: 

j.j . '"s'^e fne both 

^l, i ri^^^^^ f'n'-^ucs, the results 1 fit ^'^'^n 
^l-'tn fro,,, tl,c I, tr!, ' 'V' ^^'^P'^r, and I shli; n "'''°'P'^'^'<^ and ret- 
^l^'-^-»,s U rZjT"'''^: - "srm,.d - 
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1. Does the scheme employ a sign or category system? (A category system 
allows the observer to document a behaviour-c.g., teacher asks question- 
every time it occurs, while a sign system requires an obse.-vation to be 
made at regular intervals of time.) 

2. Does the scheme require the documentation of high or inference 
behaviours? (A high inference behaviour requires observers tc interpret the 
behaviour they observe - e.g., on-task or off-task behaviour.) 

3. Does the scheme allow a particular event to be assigned to rfiore than one 
category or event? 

4. Is the instrument intended to be used in real time or on video/audio 
recordings? 

5. Is tiie scheme intended principally for research or teacher education? 

6. What is the focus of the instrument? (Schemes can enable the researcher to 

focus on one or more of the following: verbal, pnralinguistic, nonlinguis- 
tic, cognitive, affective, pedagogical content, or discourse.) 

Since their first appearance on the research scene, observation schemes have 
bcconic increasingly .sophisticated - an inevitability, given the relative sim- 
plicity of the earlier schemes. For example, one early .scheme provided eight 
categories of teacher language (e.g., teacher explains a grammatical point/ 
teacher prai.ses/teacher criticises) and only three categories of learner lan- 
guage (learner asks a question /learner answers a question/learner talks to 
at,other learner). The insights generated by such simple schemes are 
extremely limited, and for research purposes (as opposed to teacher education 
purposes) the simple early schemes were replaced by increasingly sophisti- 
cated .schemes. 

One such sophisticated scheme is the Communicative Orientation of l-an- 
guage Teaching (COLT), %vhich was developed to enable researchers to com- 
parcdiffcrent language classrooms (Chaudron 1988 identifies 84 different cat- 
egories in the COLT scheme). 1 he aim of the scheme is to enable the observer 

to describe as precisely as possible 

some of the features of communication which occur in second language ciassrtwms. 
Our concept of communicative feature has been derived from current theories of 
cummniiicative conipctcncc, from the literature on communicative language 
teaching, and from a review of recent research into first and second language 
acquisition. The oliscrvational categt)rics are designed (a) to capture significant 
features of verbal interaction in L2 classrcxjms, and (b) to provide a means of 
comparing some aspects of classr(x)m discourse with natural language as it is used 
outside .the classr(X)m. (Allen, Friihiich, and Spada 1984: 233) 

The COLT consists of two parts. Part A focuses on the description of class- 
room activities and consists of five major parts: the activity type, the partic- 
ipant organization, the content, the student modality, and the materials. Part 
B relates to communicative features, and isolates seven of these; the use of the 
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target language, information gap, sustained sp)eech, reaction to code or mes- 
sage, incorporation of preceding utterance, discourse initiation, and relative 
restriction to linguistic form. Some of the key questions relating to each of 
these features are set out in Table 5.2. 

In Table 5.2, 1 have simplified (although, I hop>e, npt oyersimpiified) one of 
the more comprehensive of the observation schemes to have been developed. 
It is worth noting that, while the developers of COLT have tried to devise a 
set of procedures which enable trained researchers to obtain reliable data (and 
the reliability can be enhanced by the use of more than one observer or rater), 
the categories and communicative features are subjective to the extent that 
they have been selected with reference to a particular theory of language, and 
also with reference to current research. This serves to underline the point that 
there is no such thing as 'objective' observation, that what we see will be deter- 
mined, at least in part, by what we expect to see. Our vision will also be influ- 
enced by the instruments we develop, adapt, or adopt to assist us in our obser- 
vations. While the use of observation schemes can provide a sharper focus for 
our data collection than unstructured observation, it can also serve to blind 
us to aspects of interaction and discourse which are not captured by the 
scheme, and which may be important to our understanding of the classRK)ni 
or classrooms we are investigating. Later in the chapter we shall look at some 
research which utilized the COLT scheme. 

r 

\ Interaction analysis 

1 Another mefhod (or cluster of methods) for analysing classroom interaction 
^ involves the discursive analysis of classroom talk. There are numerous schools 
of thought and a range of methods and techniques for carrying out such anal- 
ysis. For illustrative purposes, I have chosen an approach adapted from the 
work of Halliday, and Halliday and Hasan, by Lemke (1985). Lemke 
embraces a social f>erspective on language that sees sch<x)ls not as 'knowledge 
delivery systems' but as social institutions in which people affect each other's 
lives. He argues that classroom education is talk: 'It is the social use of lan- 
guage to enact regular activity structures and to share systems of meaning 
among teachers and students' (p. 1). Interpreting education as the use of lan- 
guage in the context of social activity enables the researcher to observe, doc- 
ument, and interpret how teachers and students use language across all school 
subjects to bdild relationships, define roles, and so on. Two key constructs in 
Lemke's approach to discourse analysis are 'activity structures' and 'thematic 
systems'. He begins his analysis by pointing out that classroom language ful- 
fils important functions in addition to providing information. It is also used 
as a tool for social action, and for creating meaningful contexts. Lemke's aim 
is to develop a system of analysis which captures the dynamics of social inter- 
action at the same time as analysing the thematic content of the subject being 
taught. Such a system must allow for the simultaneous interpretation of the 
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TABLE 5.Z QUESTIONS REL.A 
SCHEME 



riNGTO THEPRlNt 



;|l.Al.l^EATl)KES Ol l lli; H)l 1 



Feature 



Part A; dassrown actwities 
la. Activity type 

2a. Participant organization 



.3a. Content 



4a. Student niodaiity 
Sa. Materials 



Qucitkms 



Tart B: classroom language 
lb. Use of target language 
2b. Information pap 

3b. Sustained speech 

4b. Reaction to code or message 

5b. Incorporation of preceding 
utterance 

6b. Discourse initiation 
7b. Relative restriction of linguistic 
form 



Whnt .s the .K-tivlty \y\^- c.g..,Jnll, rule 

Is the teacher working w.th the whole cL.ss 
or not? 

Are students working in groups or 

individu-illy? 
Ifuroupwork.howis it organr/.ed. 
Is the focus on cl..ssr..«m nunas^etnent, 

language (form, luiKtum,J.sc<.urM', 

sociolinguistics), or oilier? 
Is the range of topics broad or n..rrow. 
Who selects the top,c- teacher, students, or 

Are students involved in listcnu.g, spe..ki..g. 
reading, writing, or comb.iwfonsol 

these? 

What types of materials are uscU. 

How long IS the text? 

What is the source/purpose- ot the 

materials? 
How controlled is their use? 

To what extent is the t.UBCt language used? 
To what extern .s requested intornuuKm 

predictable in ailvancc? 
Is discourse extended or restricted to a 

single sentence, clause, or wore ? 
|)(h:s the interlocutor react to code or 

message? ,. 
D.H.-S the speaker incorporate the preccd.nt 
utterance into the his or her 
contribution? ■ • • , 

Do learners have opportunities to initiate 

discourse? 

Does the teacher expect a specihc form, or 
there no expectation of a particular 



llnniikrir form: 



Rrsciirih iiicthcnls in hiiti^ittif^r li\iniiiig 

iiticnince in intcrnctioiml and siihsLiiitivc terins, and thus sliow how each 
utterance contrihiitcs to tlie enactmcntof specific activity structures and the 
development of tlic thematic systems. 

l emke ilhistrates the ways in which interactional and thematic aspects of 
cl.issrooni discourse operate with reference to the following episode from a 
New York science classroom: 

1 T: IWforc wc get started . . . Ix'fore 1 erase the \niaa\ - 

2 Ss: Sh! 

3 'I': LJh - Look how fancy I pot - [ points to the board] 

4 S: I Makes a funny noise] 

5 Ss: Sh! 

6 I : I his is a rcprcsciit.itioii of the IS orMtal. 

7 S'poscil to he of course - tlirce diinensional. 

8 What two elenuiits could be represented by such a diagram? 

9 Jcniiifir? 

ir' I: 1 lydrogeii anil helium? 

11 1:1 lyiiri>i;cii anil helium. I lyilro};cii would have electron - 

12 somcwliirc in there - and liclimn would have? 

13 S: 7 If 'o electrons. 

14 T: ['wo I pause I I his is IS, and the white would Ik-? 

15 IVIark? 

16 M: 2S 

17 I': I'wo S, aiiil the green would he? uh . . .Janice 

18 I: 2!'... 21' 

19 J: 21' 

20 T: Two 1'. Yeah the green one would Ix.' 21'^: and 21'y. If I have 

21 one electron in the 21'.v, one electron in the 2I'y, two 

22 electrons in the 2S, two electrons in the IS, what element 

23 i'i Ix-iiij; reprcsi-iiteil in this conHguration? 

24 I scniching noise | Oo! that soimd annoys, doesn't it? 

25 Ron? 

26 K: Boron? 

27 T: That would Ix- - that'd have uh - seven electrons, so you'd 

28 have to have one here, one here, one here, one here, one here, 

29 one lierc. Who said it - you? 

30 S: C^arlxin 

31 Ss: ("arU)n ("arl^on 

32 T: (^arlxm. C]arN)n. Here. Six electrons. 

33 And ilu y can Ix." anywhere within those - confining orbitals. 

34 This is also from the notes from before. I hc 

35 term orbital refers to the average region traversed by 

36 ati electron. I'lectrons occupy orbitals that may differ 

37 in si/.c, shape, or space orientation. That's - that's from 

38 the other class, we might as well use it for review. 

(pp. 3-4) 
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What sense does Lemke make of this classroom episode from the perspce 
tive of a« vity structure and thematic development? The foilowmg verba,... 
extracts answer this question. 1 have included them as good examples of how 
"scu^ive analysis 'works' (we shall look in greater deta.l at the method m 
Chapter 8). 

fn'FplL'l'T o^" it gets underway, the teacher asks a question and calls on 
In t,pisoac I, oiK-i | b answer is made w th a questioning 

, He may preface a question with a brief preparatory remark (as m 

repea .ng them^ He may P ct. c q ^^^^^^^^^ ^. ^ ^ ^^^^^^ supplementary 
hues 1 1 and and he may to ^^.^ ^^^^^^^ 

Tn^You a r aX i haHt^^^^ possible to rewrite the whole dialogue 
L"::::;he:m";d^^^^^^^^^ essentially the same sc.nce con.e^^ bc.ng presented. 
,„.Hesix.aluuerac.^^^^^ 

begun with his question. (Lemke 1985: 18 19) 

IK) 15, you ca 7 between orbital diagrams that can 

anchc, in .be Unso.gc of ,1,. c .»,oon, o, ,«,b<»k. • To > 8' «> *er. 
rw.,hisi,i«i^flTlpl»,s,-uc,u,.ofb„il^^^^ 
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TABLE 5.3 A COMPARISON OF METHODS FOR CLASSROOM RESEARCH 



Melhtxi 



Typical questions 



Formal experiment 



Stimulated recall 



Observation schemes 



interaction analysis 



Is Method A superior to Method B? 
Do authentic listening texts lead to greater language gains 
than non-authentic listening texts? 

What is the basis upon which teachers decide to depart 
from their lesson plan in the course of a lesson? 

Why do teachers decide to correct some errt>rs and not 
others? 

What is the range and distribution of language functions 

by teachers and pupils in the language classroom? 
In mixed ability classrooms, how often are low 

proficiency students addressed by the teacher? 

How do teachers maintain power and control through 
classroom discourse? 



degree that is probably outside the conscious recognition of teacher or students as it 
happens. By contrast, explicit formulation of definitions and relationships is brief 
and occasional. (Lemke 1985: 20-21) 

I have included the description of Lemke's work both because of its intrinsic 
interest and xo illustrate a procedure which identifies significant patterns and 
events without assigning classroom talk to predetermined categories, as is the 
case with observation schemes. The different methods discussed in this section 
are summar9sed in Table 5.3. 



A review of research 

When one examines the literature on classroom observation and research, 
one is struck by the relative paucity of research such as Lemke's where the 
data were actually collected within genuine classrooms. (As 1 have said, by 
'genuine' classrooms, 1 mean classrooms which were specifically constituted 
for the purposes of teaching and learning, not to provide a venue for research.) 
In a recent review, I examined fifty widely cited studies from the classroom 
research literature (a report of this survey can be found in Nunan 1991b). The 
five dimensions of the analysis were as follows: 

1. the environment in which the data were collected (whether they were col- 
lected in genuine classrooms or not) 

2. the rationale for the research (whether the principal aim of the researcher 
was to provide insights which might be acted upon in pedagogy, or 
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whether it was to investigate processes of acquisition with only a second- 
ary rationale relating to pedagogy) 

3. the research design and method of collection 

4. the type of data collected 

5. the type of analysis 

The survey revealed that of the fifty studies, only fifteen were actu.illy c uncd 
Lt in genu ne language cl.ssrcxinis. A further seven collected sonic ot .iK.r 
da a m Ts^rooms'and some of their data .>nt of class. The great n.„or,.y 
coLted he r dat; outside of the classroom in lalx>ntc>ry, snnul.ucd and 
natS t c s^^ Despite this, the researchers had no hesu.n.on ,n clann- 
g"8ic relelance for tlieir research. The design w ether or not a for- 
nial Experiment was used) and the data coUec.on methods cmph.ya a.e sa 
out inTables 5.4, 5.5, and 5.6. (Note that some stt.dies ufhsed more than one 

" OnTtf the disturbing outcomes of the survey was the fact that only fifteen 
of the studies were classroom based. If context is important to researclMu.t- 
comes, then we need far more of these classr<x>ni4.sed. as opfx scd u, b^^^ 
room-oriented, studies. Such a view is strongly put by van ..er I m. 47), 
wTo aSues th t the key task for classroom researchers >s 'to .dent.fy, describe 
and relL, in intersubjecive terms, actions and contributions of parncp.m s 
fn the L2 dassroom, in such a way that their significance for language lea , - 
ng can be understood'. If this is indeed the case, t en -trncting any c.u 
action and studying it out of context "complicates rather than facilitates thcr 

'^Tn theT'^elopment of classr<K>m-based as opposed to classrc.cm.-c.riented 
research, it would be good to see the emergence of studies m which a num x 
of da a Ollection methods are employed. This would enable the re;earche 
to obtain a more complete picture of the phenomena under mvestigation. 1 > 
ex.aniple. an investigation of teacher beliefs and classrooni interaction should 
desirably include not only classroom observation, but also stimulated recall, 
tacher interviews, and teacher diaries. I would also like to see a more active 
defo dassroom practitioners in applied research. The development of skills 
n observing and documenting dassroom action and interaction part.cularb^^ 
f these foster the adoption of a research orientation by teachers to tl ei class 
00ms, provides a powerful impetus to P^f^^^^^'l!:'^^,^;;^:^ 
exemplified in the action research programs described by Nu an 9«^^^^ - ^ 
also in the introduction to classroom research for language teachers by A I 
v^^ight and Bailey (1991). However, 1 also feel that such research, particularly 
Tcarr ed out in collaboration with university-based researchers, can do more 
han act as a tool for professional development. It also has the potential to 
extend the current research agenda, as well as give researchers access to a 
greater range of classrooms than is currently encountered within the research 
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TAllI r S-4 I I ASSROOM-BASKD 
STUniKS: DI SIGN AND METHOD 



Design 


Kxpcrimcnt 


2 




Non-cxpcrimcnt 


13 


Met 111 hI 


Olvicrvntion 


7 




Trnnscript 


5 




HIicitntion 


3 




Diary 


1 




Introspection 


1 



TABl.l- <i.5 LABORATORY, SIMULATED, 
AND NATURALISTIC STUDIES: DESIGN 
AND METHOD 



Design 


r.xperinient 


13 




Non-expcrimcnt 


15 


MetliocI 


HIicitation 


21 




Interview 


5 




Transcript 


2 




Questionnaire 


2 




Diary 


1 




C;ase study 


1 


l AllLl-. 5-6 


MIXED SniOIES: DESIGN 




AND MI TIIOI) 




Design 


F.x|x'rinicnt 


3 




Non-experiment 


4 


MetlwKl 


Olxscrvation 


3 




Transcript 


2 




Diary 


2 




i'licitation 


1 




Interview 


1 




Introspection 


1 




Case study 


1 



literature. (It is tio accidetit that so much research is carried out within uni- 
versity language centres!) 

Ultimately, the central question that research of the type reviewed here 
seeks to answer is wlicthcr successful second language development can and 
does take place in tlie classroom. In his own review of the relevant research, 
which focused on outcomes rather than research methods, Ellis (1988)argues 
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that it can. While his examination of the discourse processes contributing to 
such development is somewhat tentative, he suggests that eight aspects of dis- 
course might be important (pp. 128-130): 

1. Quantity of 'intake': The amount of the target language which learners 
attend to is significant - quantity alone is insufficient. 

2. A need to communicate: This can be provided if the target bnguage serves 
as the medium as well as the target of instruction. 

3. Independent control of the propositional content: Learners have a choice 
over what is said, and part of this should be content known to the learner 
but not the teacher. 

4. Adherence to the 'here and now' principle: In the early stages at least, 
encoding and decoding is facilitated if the things being talked about are 
present in the learning environment. 

5. The performance of a range of speech acts. The learner should be encour- 
aged to use a range of language functions, and perform a variety of roles 
with the classroom discourse (for example, initiating as wctl as responding 
to discourse). 

6. An input rich in directives: Particularly in the early stages of learning, 
directives occur in familiar and frequently occurring contexts, they refer 
to the 'here and now', they are morphosyntactically simple, and, as they 
require a non-verbal result, they are more likely to count as successful 
communication than interactions requiring a verbal response. 

7. An input rich in 'extending' utterances: These are teacher utterances 
which pick up, elaborate, or in other ways extend the learners 
contribution. . 

8. Uninhibited 'practice': This refers to the right to use language without 
communicative intent, and also to the opportunity to repeat utterances 
that are meaningful to the learner'. 

These principles have the virtue of having emerged from a wide range of class- 
room-oriented and classroom-based research. This empirical, rather than 
speculative, dimension is only now becoming the norm in foreign language 
education. The principles show that research can have something positive to 
say to those looking for directions and signposts to practice. This is not tosay 
that they should be accepted uncritically by the classroom teacher. The 
important thing is for such research outcomes to be applied and evaluated in 
an informed way through action-based research on the part of practitioners. 

Classroom research: sample studies 

In this section, the process of carrying out classroom research is exemplified 
by two sample studies reported in Spada (1990). The studies are particularly 
relevant to this chapter because they utilise one of the most comprehensive 
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observation instruments yet devised - the COLT (Communicative Orienta- 
tion of Language Teaching) scheme. As we saw earlier in the chapter, this 
scheme was originally devised to capture varius aspects of communicative 
language use in the classroom. The scheme has two parts, A and B. Part A, 
derived from the communicative language teaching literature, captures 
organisational and pedagogical aspects of the classroom. Part B, which is 
intended to reflect issues in first and second language acquisition research, 
documents aspects of teacher-student interaction. The instrument has been 
used in a number of instructional contexts, including ESL for children, core, 
immersion and extended French, and intensive ESL for adults. 

Study 1: ESL study 

One of the studies described in some detail by Spada was a comparative inves- 
tigation of three classes of an adult ESL program which was run as a six-week 
intensive summer course. Each class was observed for five hours a day, once 
a week, over a four-week period. The investigation sought to determine (a) 
how different teachers interpreted theories of communicative language teach- 
ing in terms of their classroom practice, and (b) whether different classroom 
practices had any effect on learning outcomes. Studies such as these, which 
attempt to establish causal links between classroom processes and learning 
outcomes, are called process-product research, and are notoriously difficult 
to carry out. 

At the beginning of the experiment, students were given a battery of tests 
including the Comprehensive English Language Test; the Michigan Test of 
English Language Proficiency; teacher-designed tests of reading, writing, and 
speaking; and a multiple choice sociolinguistic and discourse test. 

The data yielded by the COLT observation scheme indicated that one of 
the classes, Class A, differed from the other two in a number of ways: 

A spent considerably more time on form-based activities (with explicit f(Kus on 
grammar), while classes B and C spent more time on meaning-based activities (with 
focus on topics other than language). Classes B and C also had many more 
authentic activity types than class A. Furthermore, the classes differed in the way in 
which certain activities were carried out, particularly listening activities. For 
example, in classes B and C, the instructors tended to start each activity with a set 
of predictive exercises. These were usually followed by the teacher reading 
comprehension questions to prepare the students for the questions they were 
expected to iisten for. The next step usually involved playing a tape-recorded 
passage and stopping the tape when necessary for clarification and repetition 
requests. In class A, however, the listening activities usually proceeded by giving 
students a lis: of comprehension questions to read silently; they could ask teachers 
for assistance if they had difficulty understanding any of them. A tape-recorded 
passage was then played in its entirety while students answered comprehension 
questions. (Spada 1990: 301-302) 
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The qualitative analysis confirmed the class differences shown.g, Or cx.un- 
pie, that class A spent twice as nmch time on form-based work as class C, and 
triple the time spent by class B. To investigate whether these d.tlerenccs con- 
tributed differently to the learners' L2 proHciency , pre- .ind posttrcainicni test 
scores were compared in an analysis of covariance. (You will recall Ironi 
Chapter 2 that ANCOVA is similar to ANOVA in that it is used t<, comp.ire 
differences between means, but that it can also ccmpcnsatc for any Mmuil dif- 
ferences in proficiency among subjects. Such differences are a distinct possi- 
bility when intact classes rather than randomly assigned sub,ects are nt.liscd 
in an experiment.) . , ^, . i i 

Among other things, results indicated that classes B and C nuprovc-d thcr 
listening significantly more than class A, despite the fact .ha. cL.ss A spent 
considerably more time in listening practice than tlie other classes. I he 
researcher concluded that there are measurable differences in the way in 
which instruction is carried out in classrooms, and that .l.csc d.lfcrcnccs arc 
reflected in different learning outcomes. This research demonstrates the fact 
that qualitative observation and analysis are needed in order to interpret the 
quantitative results obtained from the administrati(,n of standardised tests, n 
other words, the finding that some students did significantly k-.ter than oth- 
ers on the posttreatment tests would have k-en uninterpre.able had the 
research not obtained qualitative data on different classroom practices. 

Study 2: Core French study 

A second study reported by Spada involved eight eleventh grade core French 
classes The classes were selected because they purported to represent two 
contrasting orientations to instruction, being either 'analytical/strnctural in 
orientation or experiential/functional' in oriciuaii.m. 1 he aim ol .he 
research was to determine how these different orientations were actually real- 
ised at the level of classroom pedagogy, and whether the different orientations 
would lead to different learning outcomes. 

All the students taking part in the investigation were given a battery o pre- 
experiment and post-experiment proficiency tests, including a multiple 
choice grammar test, a writing test, and a test of listening oniprehension. In 
addition, each class was observed on four occasions during the school year, 
and the classroom interactions and behaviours were documented utilising the 
COLT scheme. These observation data were used to characterise classes as 
either Type E (experiential/functional) or Type A (analytical/structural). 

Classes were categorized as Type E or Type A by taking the t'^^'l peree'Uage c.l 
time spent on each of the experiential features in I'arts A ami U ol C.Ol. 1 (e.g.. 
group work, unpredictable language use, sustained speech, focus on topics/ 
meaning, reactions to message), adding them together for each class, and ranking 
the individual class totals. (Spada (1990: 304) 
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On this bnsis. two classes were categorised as expericntal, with the remaining 
SIX hcmg classed as analytical. Type A classes spent more time on teacher- 
controlled topics, on tasks involving minimal written work, and on form- 
tocused activities. Also, surprisingly. Type A classes spent less time in whole- 
class iiitcract.on than their Type E counterparts. Analysis on Part B of the 
COLT scheme revealed that 

students in Type F clas.scs spent a grcntcr mnount of time producing sustained 
s,.ccch, rcnctiMK to rncssngc, .ind expanding each other's utterances than students in 
1 ype A classes. In additujn, students in Type F. classes were less restricted in 
lanKuagc use than students in Type A classes. Finally, while teachers in Type A 
classes reacted siRnihcantly more to akfe than message, teachers in Type E classes 
iltil rhe reverse. (Spaila 1990: .305) 

Havin,; cstnhlishcd several statistically significant differences in instruc- 
tional treatment between the two types of classes, the researchers then set out 
to determine whether these differences resulted in different learning out- 
comes. In attempting to derive causal links between instructional processes 
and learning outcomes, the researchers first compared the scores of the two 
groups of students using analysis of covariance and found that there was no 
difference on any of the measures of proficiency. They then compared just 
two classes, one from either end of the experiential-analytical continuum. 
Here, the only significant result was that the analytical students were better 
on the gmminar test than the experiential learners. Finally, the researchers 
correlated posttreatment test scores with all categories of Parts A and B of the 
C.OI.r scheme. This analysis resulted in rather mixed outcomes. On Part A 
of the COl.T scheme, successful classrooms seemed to k- those in which the 
teacher did relatively more talking than individual students, relatively more 
tune was spent on cla.ssroom management and form-focused activities than 
on general discussion, the students themselves spent relatively little time 
speaking, and visual aids and LI materials were used relatively often. On Part 
B, it was foiiiui ihat 'genuine questions, reaction to mcss;ige and topic incor- 
poration were positively related to improvement, whereas sustained speech 
by students, predictable questions, and reaction to code were negatively 
related. These results imply that learners benefited from both the analytical 
and exjx-rieiitial aspects of instruction' {Spaila 1990: .K).9-.l()6). 

rhe studies reviewed in this section illustrate a numkr of important 
points. In the first instance, they show that collecting pre- and posttreatment 
test data is insufficient, that in addition, one requires process data on what 
actually goes on in the classrooms. The studies are also of interest because 
they are examples of process-product studies, which are notoriously difficult 
to carry out. In the first place, it is not always po,ssible to carry out a true 
experiment m which subjects arc randomly assigned to groups. Secondly, the 
treatment might not show an effect Ix-cause the length of time is not sufficient 
to show [jro/icieiuy gains. Finally, there is the problem of operations lising the 



I OS 



Classroom observation and research 



constructs one is working with, particularly that of proficiency. One [xjssible 
reason for the mixed results in the core French study is that the 'analytical- 
'experiential' distinction is more ostensible than real. It may also be that pro- 
ficiency measures selected lack construct validity [for example, it may be 
questioned whether the multiple choice grammar tests used by the researchers 
capture the notion of proficiency as the ability to carry out communicative 
tasks in the target language (Richards 1985)]. Another question concerns the 
relative contributions of form- and meaning-focused activities to the devel- 
opment of communicative competence - simply tabulating the amount of 
time devoted to different activities may be too crude as an index of analytical- 
experiential teaching. The insight that both analytical and experiential 
instruction are necessary may be only a first faltering step in the direction of 
developing effective instruction. To act on this insight, a teacher would want 
to know how much of each tyjje of instruction is appropriate for a given 
group of learners, when they should be introduced, and how they relate to 
one another. 



Conclusion 

In this chapter I have set out some of the issues and methods relevant to the 
investigation of language learning and use in classroom conte.vcs. Obviously, 
with such an enormous area, about which entire volumes can, and have, been 
written, it is only possible to provide some initial signposts which might assist 
you as you read further in the field and, it is hoped, also begin your own 
exploration of classroom teaching and learning processes. This last point is, I 
believe, an important one because: 

The teacher is the researcher's link with learners, and also the learners' link with 
research. The teacher is contracted to help learners learn, but can do so better by 
knowing about previous research and by using the procedures of classroom research 
to understand better what is happening in his or her own classroom. In this way, 
the exploratory teacher will not only improve achievement but will also contribute 
to our general research knowledge about how language classrooms work. 
(Allwright and Bailey 1991: 197) 

In addition to outlining methods and issues, I have tried to give some indi- 
cation of the problems confronting those currently engaged in classroom 
research.- Much of the research which purports to provide teachers with guid- 
ance on pedagogic practice is not derived from genuine language classrooms, 
and the research agenda needs to be extended to incorporate a greater range 
and number of classroom-based as opposed to classroom-oriented research. 
(Although, to repeat, 1 am not questioning the value of, or necessity for, class- 
room-oriented research.) 
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Questions and tasks 

1. One of the disadvantages of observational instruments is that they act as 
mental 'blinkers' on the user. They also encapsulate the author's ideological 
beliefs about the nature of teaching and learning. What ideological assump- 
tions about the nature of instruction are revealed by the authors of the COLT 
scheme? What questions or hypotheses might be investigated using schemes 
such as these? 

2. Select three or four papers relating to some aspect of teacher talk, learner 
talk, or teacher-student interaction, and carry out a comparative analysis 
similar to that carried out by Nunan (1991b) and reported in this chapter in 
the section, 'A Review of Research'. 

3. Study the classroom extracts and stimulated recall statements presented 
in the first section of this chapter. What research questions and issues do they 
suggest? Alternatively, carry out a similar task using the extract in task 4 
below. How might these research questions and issues be followed up? 

4. a. What similarities and differences are there between Ellis's list of eight 
conditions for successful language development and the conditions implicit 
in the COLT scheme? 

b. Study the following classroom extract (author's data) in terms of the ques- 
tions from the COLT scheme as set out in Table 5.2. (You may not be able to 
complete ail categories.) 

c. To what extent does the teaching/learning sequence reflect the following 
eight discourse conditions hypothesised by Ellis (1988) to be important in fos- 
tering second and foreign language development? (These principles are 
glossed in this chapter.) 

(1) quantity of intake 

(2) a need to communicate 

(3) independent control of the propositional content 

(4) adherence to the 'here and now' principle 

(5) the performance of a range of speech acts 

(6) an input rich in directives 

(7) an input rich in 'extending utterances' 

(8) uninhibited 'practice' 

Transcript 

Step 1: Teacher puts students into small groups and gives out a handout containing 
the following information: 

Student A: What helped me most to learn English? Let's see - reading all sorts of 
printed material, listening to native speakers on the radio, TV and films, finding 
opportunities to use English out of class. 

Student B: The things that helped me least - well, 1 would say memorizing 
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grammar rules, reaiiiiig alouti one by one arouiui tlic ci.iss, doiiij; lx)riiig 
grammar exercises. 

Student C: Language taught inside the classriKnu is not suiticieiu id iii.ike a 
person a competent speaker in the real world. You need to use the l.irigu.igi' 
outside of the classr(X)ni. 

Student D: Practising through conversations and using tiic malia, es[jeilally TV 
with subtitles and newspapers. You must h.ive someone who is proticiciu in the 
language to speak with in order to learn the language sufficiently well. 
Student E: 1 find that motivation is vital in the suece.ss of learning a foreign 
language. Strong interest, sheer determination and motivation to learn a seeoiul 
language were the most important things for me. 

Student F: I would say 'teacher talk' helped me least, looking back, I wish he 
had given me more opportunities to use the language in class, es|x'cially 
sp>eaking it inside and outside the elassnMim. It would have Ix-en more fun and 
challenging if I was thrown into the deep end. 

Student G: The thing 1 liked least was negative oral criticism and punishment 
for wrong answers. Dull teachers who are iiiactive/caniiDt Ix' lie.ird cie.irly. 
Student H: What helped me most was constant drilling, and when I li.ui my own 
textbook and made notes from teacher explanations. 

The teacher tells the group that some additional .students will Ix- joining the class. 
The handout contains .statements from these students. The stuilents li.ive to decide 
which of these additional students they would most like to join their group, ami 
which they definitely would not like. 

Step 2: Small group discussion. The students spend 25 minutes completing the task. 
Step 3: Debriefing. l°he teacher leads a whole group discussion. 

T: All right, this group has finished so we can all stop. What I'd like to do is get a 
short oral summary of the discussion you had in your group - what responses 
you came up with in each group. I'd like to start with the group down the 
back. G. 

S: Our group strongly agree with A, C, and er E. And we disagree or neutral with 
the rest of the student. Definitely we will havc/hate(?) to join our group the 
student H . . . 

T: Which one was that? Let me see. 

S: The last one. 

T: You would like that person to join your group (no we don't). You don't want 

that person in your group (no). Mmm. 
S: We would like to, to join to our group the student C - and if [x)ssil)le to have 

student A as well. 
T: So you have a back up - a reserve. Why did you choose student C? 
S: Because we think that is more er dynamic, that meth(xl the student is using - 

you know, is more . . . more aggressive, more er . . . come on, give me help 

(laughter). 

T: All right, you've done very well. Did anybody in your group disagree, or did 
you all agree on the fjoints you discussed? (yes, we agree/ we agree) I suppose 
that's why you're sitting togethet, it is? (laughter) 

T: Which group can I pick now? What about this group over here? 
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S; C.Mt I ask you one question before 1 answer any question? 
T: Stire. 

S: Do wc suppose to only choose one person as er dis . . . agree or disagree - or 
have more . . . 

T: Well, I cr I he instructions were pick one (only one) pick one student 

S: That's it, yes. i our group, wc arc most agree with the student E, and we are 

disagree with student F. 
T: So you'd like student E to join your group and you would not like student F to 

join your group (exactly). Can you give . . . can you tell me the reasons that you 

decided? 

S: Uin the reason lieing is because the student F. um mention very clearly the most 
important aspect - that is what is important for learning - and student F was 
very narrowcil the subject and just point the negative point of view of learning, 
so . . . (mm) 

T: OK Docs anylvniy want to ask this group any questions? No? OK, this group. 

S: 111 our group, we would like student F, to join our group. 

1: Student F? (JDlly, two votes for student I", so far. Mm 

,S: "i'oii want to know the reason why? (yeah) Yeah. Because wc think that the 
stiulcnt got a motivation to learn something cr wc can pay attention to it and 
then we can progress quickly, (yeah, OK) Er, we dislike student A to join our 
group (student A) yeah 

T: Whyisth.u? 

S: i-.r, wc feel the student, he pay attention too much on er hearing, but he did not 
pay attention to writing English - written English. Not much time to u.sc er 
written English. 

T: OK, right - Kigiit . . . this group. Come on, don't be shy (laughter) 
S: Actually wc agree with student A . . . 

I': Student A isn't that good, great, we've got one group at least that one group 

would like student A and one group would hate student A in their group. 
S: And, er, student D is strongly agree (.student?) D(D) strongly agree. Yes. 
T: Mmni. Rut, but you'd like student A to join your group. 
.S: No. I) 

T: Student O? (D) D? Sorry (yeah) And you wouldn't like student A to join your 

group, (iiiaud) So which is the one you, you dislike? 
.S: Dislike? Student I I. 
.S: Student 1 1. 

T: Yeah, why? 
S: It's er neutral 

T: {Mni)C;an you give the reason why? 
•S: We'll have an headache, if he join to us (laughter), 
I': OK - tills group. 

.S: Uin, the student which wc most like to join the group is student C (Mm) And 

the one which we don't like to join ilie group is student F. 
T: All right. Can you give - can you tell me the reasons? 

S: Well studctit ( ' - er, reason why we like him to join the group is because he's 
looking at the practical - at the practice of language outside the classroom. And 
we think - we all agree that it is very important for the student to practise the 
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language. So, he learn the language inside the classroom, and he go outside and 
practise it, and that's what help him to learn English. 
T: Right, and the negative one? 

S: Er, the negative one - I, I think the teacher the teacher talking too much it docs 
help the student because they're learning English. Secondly, he saying that the 
teacher didn't allow him to practice English in, in the classroom - well there is a 
limit to that because she can't give everybody all, she can't give one person all 
the time she have - she has to give an opportunity to the rest of the student. 
Secondly, er, thirdly, he said that she didn't allow him to use the language 
outside the classr(K)m but she had no control over him outside the classroom - 
he can go and do whatever he like. So he's blaming the teacher for that. 

T: I'm sure all the teachers in the room will love you. (laughter). OK. Uni. 

S: Er, the student we like most is E (E) Yeah We dislike B. Because as to E wc 
think er it's (E yes) very important for learning a language its interest and 
determination and motivation that if you cr - if you have strong strong interest 
and motivation then you will try to find the how to learn a language is very 
creative, you will find a creative way to (uhuh) learn a language. 

T: OK (yes) Good. Did you all agree, or did you have to discu.ss it for a long time 
licfore you (we discuss) OK fine, thanks. And, last, but not least . . . 

S: We, cr, we disagree with student B, strongly disagree (uhuh) because er he 
thinks that grammar - memorizing grammar rules and doing boring grammar 
exercise - exercises - is cr not useful, that helped him least. So, wc think er the 
grammar, is a construction of language . . . that if you want to explain anything 
and you know every word in the dictionary, you don't want er what can be. er 
what order it must be used - in what order it must be used. 

T: OK. .So that's the negative - that's the student you don't want. (Yes) Which 
.student do you want? 

S: Wc chose er student H which who is cr constant drilling and er doing exercises 
with teacher who explain him if he's correct or not and why - so cr I think wc 
cr we have experience in this way of learning so we chose it. 

Step 4: Conclusion. The teacher spends five minutes introducing students to the 
idea that different learners have different preferred learning styles and strategics, 
and describes how a focus on learn ing-how-to-learn will be an important aspect of 
their course. 

5. a. What do you see as the key con.structs underlying the investigations 
reported by Spada? What do you think might be the chief threats to the valid- 
ity and reliability to the research? How might these be guarded against? 

b. Spada argues that the qualitative and quantitative data arc complemeii- 
tary. What evidence is there in the descriptions of the .studies to support this 
claim? 

c. What criticisms, if any, would you make of the studies and the conclusions 

reached by the researchers? 

6. Select a classroom-based study from a recent journal such as Language 
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Learning, TESOL Quarterly, or the Modern Language Journal. Summarise 
the study by providing the following data: 

a. Question/hypothesis 

b. Significance/value of the study 

c. Subjects 

d. Procedure 

e. Type of data 

f. Type of analysis 

g. Conclusions 

h. Further research 

i. Critique 

Further reading 

Allwright (1988) is an excellent annotated collection of important papers on 
classroom observation. Van Lier (1988) puts forth the case for an ethno- 
graphic approach to the analysis of language classrooms. A very different and 
therefore complementary perspective is provided by Chaudron (1 988). Nunan 
(1989) provides a guide to classroom observation and action-based research 
for the practitioner who may not have had formal training in research meth- 
/,oa '^"'^"^ thirty-three methods comparison studies, see Beretta 
/ oo^ ■ n ^^"'^'^'^ by Brumfit and Mitchell (1990) and Nunan 
( 992) illustrate what can be achieved through small-scale, action-oriented 
classroom research. Frolich, Spada, and Allen (1985) and Spada (1990) exem- 
plify larger-scale classroom research. 
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Here is Edward Bear coiniim downstairs now, bdinp, hump, 1hiiii|j on die hack ot 
his head . . . ic is, as far as he knows, the only way t)f coniiii}; downstairs, hut 
sometimes he feels that there really is another way, if only he could stop hiimpini; 
for a moment and think alx>ut it. 

(A. A. Milne, The World of VoM 

One of the problems confronting the iaiiguage researcher is that a grent deal 
of the hard work involved in language development and use is invisible, goinj? 
on in the head of the learner. During the days when behaviourist psychology 
dominated language research, it was considered Ixith futile and irrelevant to 
investigate this invisible work, the researcher being interested only in the 
observable characteristics of human behaviour. These tlays, however, such 
research is generally considered inadequate, and the focus of attention is 
much more on the cognitive processes underlying human perforniante and 
ability. It is widely accepted that if we want to understand what people do, 
we need to know what they think. Researchers often go to considerable 
lengths to derive insights into the mental processes underlying observable 
behaviour, and in this chapter, we shall l<x)k at the intro.spective techniques 
which are designed to help them do this. 

Introspection is the process of observing and reflecting on one's thoughts, 
feelings, motives, reasoning processes, and mental states with a view to deter- 
mining the ways in which these processes and states determine our behaviour. 
The tradition of using introspective methods has come from cognitive psy- 
chology, where it has aroused considerable controversy. In fact, many psy- 
chologists assume that verbal reports have nothing at all to do with causal 
cognitive processes. Its use in research into language learning is more recent, 
although it is certain to arouse a similar amount of controversy as it becomes 
more widely used. Particularly contentious is the assumption made by 
researchers that the verbal reports obtained through the introspection carried 
out by their subjects accurately reflects the underlying cognitive processes giv- 
ing rise to behaviour. In other words, there might be a discontinuity between 
what the subjects believed they were doing and what they were actually 
doing. 

In this chapter, I shall use the term introspection to cover techniques in 
which data collection is coterminous with the mental events iK'ing investi- 
gated. I shall also use it to cover research contexts in which the data are col- 
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iccrctl retrospectively, that is, some time after the mental events themselves 
have taken place. The amount of time wiiich elapses between the mental 
event and the reporting of that event inny distort what is actually reported, 
and one needs to be aware ol this when actually using one of these techniques. 
(In t)nc sense, all the techniques rejiortcd here arc retrospective in that there 
will always he a gap, however sligiit, hctwccn the event and the report.) In 
this chapter I shall consider the following questions: 

- What lio researchers mean hy introspection and retrospection.-' 

- What techniques arc tiicrc for obtaining introspective and retrospective 
data? 

- Which introspective research tools are particularly appropriate for inves- 
tigating language learning? 

Some early introspective methods 

The earliest introspective studies involved free association tasks in which sub- 
jects were requireii to say the first worii they thought of in response to a stim- 
ulus word. This recliiiiquc was widely used by psychiatrists such as Jung 
(1910). Another technique, rcporteil by Ericsson and Simon (1984, 1987), 
invoivcil getting subjects to respontl as quickly as possible if they did or did 
not uiulerstand cue sentences which were either ambiguous or difficult to 
inter|)rct. They cite as an example the sentence: 'We deprecate everything 
that can be explained", which was used by Buhier (1951) to investigate pro- 
cesses of comprehension. The rcsearciicr would read out sentences such as this 
and the subjects were required to indicate immediately whether or not they 
understooil. 

These techniques were fraught with problems and aroused considerable 
controversy. Ilricsson and Simon (1987) argue that techniques such as free 
association aiul sentence recall are not particularly useful for developing mod- 
els of the mental processes underlying performance, because there is no way 
of knowing whether a subject's responses were determined before the prcseii- 
i.iiion ol tiic task. It is also possible that the response may have been deter- 
mined by a procedure which is irrelevant to the task. In free association, any 
resfjonsc is acceptable regardless of how unusual or even peculiar it might be. 
These early tasks assumed that subjects were reliable, and that they could be 
trustctl to follow the procedures and instructions deviseil by tlie researcher. 
More recently, techniques have been devclojx;d in which the trustworthiness 
or otherwise of the subjects and the status of their response are irrelevant 
because the researcher decides the correct solution or answer in advance. 

Kcccin stiulics of tliitiking have prifnnrily iiscJ t.isks in logic . . . mathcin.itics . . . 
prohahility . . . aiui so on, where ;i given t;isk has a single correct answer. Using 
tasks in a fonnalizeil doniatn has many advantages. It is easy to generate a large 
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rumkr of different problems among which only the surface elements differ. In 
jJJition, the investigator can make a careful analysis of the task, which may 
.ugpcst what kinds of theory may be reasonable before observations of people 
performing the task are gathered. (Ericsson and Simon 1987:.27) 

Think-aloud techniques 

Think-aloud techniques, as the name suggests, are those in which subjects 
complete a task or solve a problem and verbalise their thought processes as 
they do so. The researcher collects the think-aloud protocol on tape and then 
analyses it for the thinking strategies involved. The following .s a transcript 
of a think-aloud protocol from a subject mentally multiplying 36 times 24 
reported in Ericsson and Simon (1987: 34). ^ 

OK 

?6 times 24 
urn 

4 times 6 is 24 
4 

carry the 2 

4 times 3 is 12 
14 

144 
0 

2 times 6 is 12 

2 

carry the 1 
2 times 3 is 6 
7 

720 
720 

144 plus 72 
so it would be 4 
6 

864 

The think-aloud technique iscloser than retrospective techniques to the men- 
tal action in which subjects think back on actions performed at some prior 
time (for example, when they are asked to talk about why they gave certain 
answers in a standardised test). However, we may still question whether he 
verbalisation accurately reflects the mental processes ^^^'^^^^^^.^''^ 
lie such problem-solving tasks. It may well be that the act of spelhng out our 
bought processes alters those processes. This worry ,s not peculiar to intro- 
spectK)n. 1 have already pointed out that researchers, ^^8^,^ f °^ ^'^^ "^'j'^^ 

5 they use or the traditions to which they adhere, will (>f they are honest) 
often have to confront the possibility that their results are in some ways arti- 
facts of the procedures they have used. 
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Anagram tasks 

A similar technique, but this time related to letters and words rather than 
numbers, is the anagram task. An anagram is a word or phrase whose con- 
stituent parts have been rearranged, resulting in 'nonsense' words. Here, sub- 
jects are presented with an anagram, or nonsense word, and are required to 
think aloud as they unscramble it to make a meaningful word. Table 6.1 pre- 
sents transcripts of three think-aloud protocols from subjects solving the ana- 
gram NPEHPA. (A protocol, as you can see from Table 6.1, is a written record 
of what the subjects say as they complete the task.) 

What do the protocols in Table 6.1 tell us about the processes involved in 
mental computation, and what sort of generalisations or hypotheses, if any, 
would data such as these enable us to formulate? According to the researchers, 
two types of strategies are frequent in anagram tasks. The initial strategy is 
to draw on long-term memory to select likely combinations of letters - that 
is, combinations that occur frequently in English - and use these to generate 
words containing these combinations. This provides a basis for attempts at 
generating a list of possible solutions. 

These can derive from attempts to sound out letter combinations or can be related 
words evoked from LTM [long-term memory]. . . . These prot<x:ols depend heavily 
upon recognition processes and evocation of information from LTM. A computer 
model couid be programmed to produce qualitatively similar protocols, but it is 
impossible, in the absence of detailed knowledge of how subjects have information 
stored and ::idexed in LTM, to predict the sequence of events in any particular 
subjects' th:inking-aloud protocol. In spite of the use of common processes, different 
subjects arrive at the anagram solution along different routes. (Ericsson and Simon 
1987: 48) 

While these think-aloud techniques represent an advance on earlier, rather 
crude techniques such as word association, they still suffer from a number of 
problems. For example, the question remains as to whether the verbalisation 
process accurately reflects the cognitive operations underlying performance, 
or whether these are somehow transformed by the subject's efforts to verbal- 
ise. The relevance of some of this research for language learning, coming as 
it does from a particular area of cognitive psychology, might also be ques- 
tioned. Nevertheless, we need to acknowledge the contribution to research 
methodology made by cognitive psychology, a contribution which may yet 
provide a significant resource for language researchers. 



Diary studies 



Diaries, logs, and journals are important introspective tools in language 
research. They have been used in investigations of second language acquisi- 
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tion, tcacher-lcarncr interaction, teacher education, and other aspects of lan- 
guage learning and use. Diaries can be kept by learners, by teachers, or by 
participant observers. They can focus either on teachers and teaching, or on 
learners and learning (or on the interaction between teachers and learners, or 
Ixrtwccn teaching and learning). In this section, we shall look at the advan- 
tages and disadvantages of using diaries as research tools in language acqui- 
sition and development. 

Bailey (1990: 215) defines the diary study as 'a first-person account of a 
language learning or teaching experience, documented through regular, can- 
did entries in a personal journal and then analyzed for recurring patterns or 
salient events'. She points out that diary studies are part of a growing body 
of literature on classroom research. As with the classroom observation check- 
lists which we looked at in the preceding section, diaries can be used either 
for research or for teacher education. While our principal point of focus is 
research, it is worth noting in passing the following benefits which can accrue 
when using research in teacher education programs (based on Porter et al. 
1990): 

1. Students can articulate problems they arc having with course content and 
therefore get help. 

2. Diaries promote autonomous learning, encouraging students to take 
responsibility for their own learning. 

3. By exchanging ideas with their teacher, students can gain confidence, 
make sense of difficult material, and generate original insights. 

4. Keeping journals can lead to more productive class discussion. 

5. Students arc encouraged to make connections between course content and 
their own teaching. 

6. Journals create teacher-student and student-student interaction beyond 
the classroom. 

7. By matching training methodology with second language teaching meth- 
odology, they make a class more process oriented. 

By extension, most of these Ixrncfits can also be applied to second language 
learning. 

In using diaries for research, Bailey (1983, 1990) and Bailey and Ochsner 
(1983) recommend a five-stage procedure, beginning with an account of the 
iliarisr's personal learning history. The data collection should be as candid as 
possible, despite the possible embarrassment of some of the entries. This initial 
database is then revised for public consumption. Patterns and significant 
events are iilentificii, and those factors which appear to be important in lan- 
guage learning are interpreted and discussed. In carrying out a procedure 
such as this, it is probably a good idea to avoid analysing and interpreting the 
data until a sulistantial amount of material has been collected. This can help 
the researcher avoid coming to premature conclusions which may be inac- 
curate or incorrect. In addition, it is often the case that in the early stages the 
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entries do not make a great deal of sense, and patterns emerge only in the 
longer term. 

It is probably not surprising that most of the diary studies in the second 
language acquisition literature are by applied linguists and language teachers 
documenting their own attempts at learning a second or foreign language. 
Most of these studies provide valuable insights into both social and psycho- 
logical aspects of language development, and in most of them affective factors 
emerge as being particularly significant in language learning. Bailey (1983), 
for instance, kept a diary of her experiences in learning French. To her sur- 
prise, the affective factors of competitiveness and anxiety emerged as highly 
significant in the learning process. 

The importance of affectivity also emerges in a study by Schmidt and Frota 
(1985). This case study documents the attempts of one of the authors, a native 
speaker of English, to learn Portuguese while in Brazil. The study is interest- 
ing for several reasons, not the least of which is the light it throws on the 
interaction of 'in class' and out of class' experiences on language 
development. 

The study consisted of three phases. In the first phase, the subject, 'R', 
attempts to pick up a little Portuguese while interacting with native speakers. 
Extract 1 from his diary illustrates this particular phase. In the extract, one 
is immediately struck by the importance of social, interpersonal, and affective 
factors to R in his initial attempts to make sense of the language. 

EXTRACT I 

Journal entry. Week 1 

I hate the feeling of being unable to talk to people around me. I'm used to chatting 
with people all d.iy long, and I don't like this silence. Language is the only barrier, 
since it is certainly easy to meet Brazilians. I've noticed that it is acceptable to ask 
anyone on the street for a cigarette. It . . . appears to have no relationship to age, 
sex or class. List night an attractive and obviously respectable young woni.in, 
accompanied by her boyfriend, stopped me and bummed a cigarette. If I take a pack 

to the beach, it disappears within an hour, so that's 20 people I could have met 

T<xlay P and 1 were at the beach, a guy came up for a cigarette, sat down and 
wanted to talk. He asked if I were an American and I said s/'m. He said something 1 
didn't comprehend at all, so I didn't respond. He said, "well, obviously 
communication with you would be difficult" (I did understand that, though 1 can't 
remember any of the words now), and left. (p. 242) 

Phase 2 of the study occurred when R enrolled for formal instruction in Por- 
tuguese. Once again, affective factors emerge as particularly important for R, 
as can be seen in Extracts 2 and 3. 

FXTRACT 2 

Journal entry. Wcci- 4 

P and 1 started class yesterday. There are 1 1 in the class (of various nationalities). 
The teacher is young and very good. She introduced herself to us (in Portuguese); I 
am X, my name is X, I am your teacher, I am a teacher, 1 am a teacher of 
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Portuguese, I'm alsoa teacher of English, I'm from [place], I'm single, I'm not 
married, I don't have children, I have a degree in applied linguistics, etc. She went 
around the class, asking the same kinds of questions: what's your name?, where are 
you frt)m?, what kind of work do you do?, do you have any children?, etc. Most of 
the students could answer some of the questions, e.g., 1 know what my title is at the 
university. Everyone was rapidly picking up new things from the others' answers. 
For the rest of the class, we circulated, introducing ourselves to each other and 
talking until we exhausted the possibilities. At the end of the class, X put the 
paradigm for SER on the board, plus a few vocabulary items. Great! This is better 

than bom dia and then silence I'm sure I'll be asked all those questions 

thousands of times before I leave here. So I went out last night and talked to four 
people. It worked, and I'm invited to a party tomorrow night. Of course I quickly 
ran out of things to say and quickly stopped understanding what people said to me, 
but that just makes me eager to get back to class, (p. 243) 

EXTRACT 3 

Journal entry, Week 4 

A half hour into the class, X showed up and pulled me out. There's a new section of 
the intensive course that's just opened and if 1 want I can move. I said yes . . . and 
went ro the new class, which was already in session . . . When I sat down, a drill 
was in progress. SER again, which must be every teacher's lesson one. Teacher asks, 
student responds: Voce e americanaf[" Are you an American/"); Sou, sini ["I am, 
ycs"J. When it was my turn the question was Voce e casado^ [" Are you married?"]; 
so I said nao. L corrected me: sou, sim. I objected: en nao sou casado. L said [in 
English], "We are practicing affirmative answers." I objected again, I'm not 
married, and L said, "These questions have nothing to do with real life." My bkxxl 
was boiling, but 1 shut up. The remainder of the class was choral repetition of the 
first conjugation verb FALAR: falo, fala, falamos, falam, over and over. I didn't like 
that much either, and when it was my turn to perform individually 1 tried to put 
the forms in sentences: en falo portugues ["I speak Portuguese"], voce fala ingles 
f'you speak English"]. L did not appreciate that at all. What a sour start! But I 
think F will stick with it. There are only three students in the class, and the other 
two are as eager as I am . . . This class will give me almost twice as many hours a 
week as the other one, and L says we will cover about twice the material I would 
get in that course. So 1 guess I can remember that I am not the teacher here, try not 
to provoke L too much, and make the best use of the resources that I get. (pp 243- 
244) 

In the third and final phase of the study, R abandons the formal classes and 
attempts to manage his own instructional program. His focus is now on 
acquiring the target language through interacting with native speakers, and 
then reflecting on that interaction. Extracts 4, 5, and 6 illustrate the strategies 
he adopts, and his reactions to success and failure in out-of-class interactions. 

EXTRACT 4 

Journal entry. Week 1 1 

H and I ate dinner at Caneco 70. He complained non-stop about his job. I tried to 
say "you don't seem comfortable" with the job: sinto que voce nao estd comfortdvel, 
and his face showed complete non-comprehension. I grabbed my dictionary. 
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"Comfortable" is comfortavel, but it fl.ished through my mind ih.ii porli.iijs you c 
only say chairs are comfortable, not (x:t)()lc, A lew niimiio l.iior 1 1 .s.iii.1 somcthinjj 
with nJo deiv. I was taught DEVER as "have to" or "must," and I've Iven 
thinking that ndodeue + Verb would mean "don't have to" liiuI deiv nao -V Vci 
would mean "must nor," but l l's remark obviously me.iiu "sliowiJ iu>t." So 1 
learned something, but in general H is a terrible conversatioii.ilist for me. He 
doesn't understand things I say that everyone else understands. When I tlon't 
understand him, all he can ever do is repeat, (p. 246) 

EXTRACT 5 

jourttal entry, Week lH 

Ijst night 1 was really up, si'lf-conhdent, feeling fliieni ... At one [x)iiit, M s.iid ic 
F that she should speak more slowly for mc, but 1 said no, please ilon't, 1 ilon't mx 
it anymore, 

KXTRACT 6 

Journal entry. Week 20 

Last night I met X, who's just come b;ick from Argentina. Before we wen- 
introduced, 1 overheard M and U talking to X about me at the othi-r eiul of tlie 
tabic. X: ele fala protugues? ["Docs he speak Portuguese?"); U: fila >nal\ "Me spi-al 
p(K)rly"]. M said 1 make lots of mist.ikes, and nientioiR-il rnarula ami />ais. X s.uv 
me UK>king at them and said: ntas voce cntende tiulo?\"\\m you umlersiaiul 
everything?"] I was annoyed and wanted to let them know I had lx.'en listening, so 
replied: entendo ntal tamhem ["I also understand pcwrly"]. 

These extracts provide a flavour of the study as a w1k))c. They ;ilst) illu 
trate the ways in which diary and journal entries provide insights into pn 
cesses of learning which would be difficult, if not impossible, ro obtain in an 
other way. The fact that R is a language teacher and researcher, as well as a 
applied linguist, also means that he has definite ideas on what is worth recon 
ing, and also on how his observations might Ik- interpreted. ()l course, tli 
question arises as to the extent to which the observations and conclusioi 
made by Schmidt and Frota can be applied to second language learners wh 
are not language teachers and researchers. 

Diary studies face problems and threats similar to other introspective dat 
collection methods. In terms of external validity, critics of the nuthod as 
how conclusions based on data from a single subject can possibly be extraf 
olated toother language learners. Such critics would probably accept tliatdi;i 
ties are ground-clearing or hypothesis-raising preliminaries to real rcsenrcl 
but that they are not of themselves valid or reliable means of dt)ing researcl 
Another criticism relates to the status of the data, and the interpretation 
derived from them. Here, the essential critical question is: To what extent di 
the diary entries realistically reflect what was really going on at the time rh 
recordings were made.' 

Notwithstanding such criticisms, it is difficult to see how the .sort of dat; 
yielded by diaries and journals could be collected in any other way. F.veti ; 
cursory reading of the diary extracts provided by Schmidt and Frota revea 
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ricli insijjiits into some of the psycliological, social, and cultural factors impli- 
cated in language development (see also Rivers' 1983 diary account of her 
attempts to learn a sixth language). 1 believe that they also have a great deal 
of potential for the investigation of learning strategics and learning prefer- 
ences of second language students (sec, for example, O'Malley and Chamot 
1990; Oxford 1990; Willing 1988). Even as a ground-clearing preliminary to 
psychometric rc.scarcli, diaries, logs, and journals have a valuable place in the 
overall methodological repertoire of the language researcher. 

Retrospection 

As I indicated at thclx-ginning of this chapter, retrospective data arc collected 
sonic time after the event under investigation lias taken place. Retrospection 
has Iven criticised by a nuinlicr of researchers (sec, for example, Nisbctt and 
Wilson 1 977) on the grounds that the gap between the event and the report- 
ing will lead to imrclinblc data. It has also K'cn claimed that if subjects know 
they will be rc(|inrcd to provide a retrospective account, this will influence 
tlieir performance on the task. Ericsson and Simon (1984) argue that the reli- 
ability of the data can be enhanced by ensuring that the data arc collected as 
soon as possible after the task or event has taken place. If subjects arc provided 
with sufficient contextual information, the reliability will also be enhanced. 
.Steps should also Ix: taken to ensure that subjects do not make inferences 
wliiL-h go Ix-yond the task, and that researcher bias is eliminated. (In inter- 
views and stimulated recall sessions, the researcher may fall into the trap of 
'leading the witness'.) Fianlly, where possible, subjects should not be informed 
tiiat they will Ixr required to retrospect until after they have completed the 
task. 

Given tiic problems associated with retrospection, one may question why 
the technique should Ix" used at all. Unfortunately, there are occa.sions when 
it is simply not tcasiblc or desirable to collect data from subjects during the 
task ()crf()rmancc. (Consider the research on teacher decision-making which I 
referred to in the preceding chapter, where the focus of attention is on the 
ilccisions teachers make in the course of teaching. In this instance, it would 
siin[)ly not Ix* feasible for the teachers simultaneously to teach and at the same 
time to report on their thoughts and decisions. Researchers have therefore 
Ix'cn forced to use a stimulated recall technique, in which they observe and 
rccorti tlic lesson. Imniedi.uely after the lesson, the teachers arc interviewed 
and prompted to talk alx)iit the decisions they made during the course of the 
lesson. Tbc lesson is suliscquently transcribed, and the retrospections are 
matched ag.iinst the relevant parts of the transcript. Table 6.2 illustrates the 
database resulting from this process (this is from my own data). 

In the study referred to in Table 6.2, Ericsson and Simon's 'immediacy' 
conditicjii was fulfilled by collecting the data immediately after the lesson. 
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TABLE 6.2 TRANSCRIPT AND RETROSPECTIVE DATA FROM A TEACHER 
DECISION-MAKING PROJECT 



Lesson transcript 



Retrospection 



T: What's the question you can ask for 

smokes? 
S: Are you smoke? 
T: Are you smoke? 

Do you smoke, or does she smoke? Does 

she smoke? 

Wh.it question docs the interviewer ask? 
The interviewer? 

What question does the interviewer ask? 

What's the question in here? 
S: You smoke? 
T: You smoke? You smoke? 

That's not a proper question is it really? 

Proper question is do you smoke? So he 

says 'you smoke?' 

We know it's a question because . . . 
why? You smoke? . . . 
S: The tone. 

T: The tone ... the ... the .. . what did 
we call it before? 

You smoke? What do we call this? 
S: Intonation. 

T: Intonation. You know by his intonation 

- it's a question. 
T: Drinks. What's the question? Drinks. 

Do you drink? 

Yeah. What's the, what's the question 
he asks? 
S: Drink? 

T: Drink. Just one word. Drink? 
How do we know it's a question? 
Intonation. 



[The students have listened to a 
taped interview in which the 
interviewer asks someone about 
their personal habits. While the 
objective of the lesson is for the 
students to extract key 
information from the interview, 
the teacher at this point contrasts 
the (native Sfxraking) interviewer's 
questions with 'proper' questions.] 
In the post-lesson retrospection, 
the teacher refers to this incident 
as one of the spontaneous 
decisions she made. 
R; So you hadn't actually planned 

to teach that? 
T: No, 1 hadn't. I mean, really, 
that would be an excellent 
thing to do in a follow-up 
lesson - you know, focus on 
questions. 
R: In fact, what you're asking 
them to do in their work is 
focus on the full question 
forms, and yet in the tape 
they're using a . - . 
T: Wasn't, yeah. So, I suppose it's 
recognising one question form 
by the intonation, then being 
able to transfer it into the 
proper question 'Do you 
drink?' rather than 'Drink?' 



Key: T = teacher, S = student, R = researcher 

The technique could have been extended by providing the teacher with the 
transcript and getting her to reflect on it, and provide a further commentary 
on the action (this, of course, violates the immediacy condition, as there is an 
inevitable gap between the time of the recording and the provision of a tran- 
scription). It is worth noting that at one point during the retrospection the 
researcher almost fell into the trap of 'leading the witness when he alluded 
to the contradiction inherent in the use of authentic language on the one 
hand, and the assertion that such language is not 'proper on the other. 
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The data yielded by this retrospective way of investigatinp teacher deci- 
sion-making gave rise to a number of inipt)rtant questions wliicli were pur- 
sued in follow-up research. These included: 

- How much decision-making is reflective and how much occurs as a spon- 
taneous reaction to events occurring in the classroom? 

- What theories of language learning and teaching underpin spontaneous 
decisions? 

- What is it that teachers focus on (e.g., the instructional process, students' 
characteristics, student behaviour) when making decisions? 

- What biographical variables (such as teaching experience and professional 
qualifications) correlate with different types of decisions? 

- What is the nature of the linguistic data which are made available to learn- 
ers, how are they made available, and to what pedagogic effect? 

Another situation in which it is neither feasible nor desirable to collect data 
during task performance is when one is investigating learners' test-taking per- 
formance under genuine test conditions. In such a situation, it is necessary to 
use retrospection. The technique is illustrated by Aslanian (1985), who inves- 
tigated the reading comprehension strategies of her learners in a second lan- 
guage context. She set out to discover, among other things, the extent to 
which responses on objective tests accurately reflect a learner's understanding 
of written texts in the target language. Her procedure was quite straightfor- 
ward. The subjects read a passage in the target language, and then completed 
a multiple choice test. The researcher then interviewed the students, and 
encouraged them to think about and report to her the reasons why they had 
given particular responses. She concluded from her research that multiple 
choice tests do not always accurately reflect the language processing abilities 
of the test-takers. While some of her subjects gave correct answers without 
adequately comprehending the text, there were others who got the answers 
wrong, but whose introspections revealed that they had, in fact, compre- 
hended the intentions of the writer. 

Feldmann and Stemmer (1987) use a similar technique to investigate learn- 
ers' test-taking performance. Their study is of interest because they use a com- 
bination of introspection and retrospection. The purpose of the study was to 
inves'tigate the operations and processes used by foreign language learners 
when working on a foreign language test. The test they chose to investigate 
was theC-test, a type of gap test in which only parts of the word, rather than 
the whole word, are deleted. The researchers wanted to investigate the con- 
struct validity of this relatively new type of test. In other words, their aim 
was to find out what the test really measures. 

The procedure followed by the researchers was as follows: Subjects (native 
speaking Germans learning French or Spanish) were tested individually. Hav- 
ing been introduced to the test, they were left alone in the room to complete 
the C-test. While completing the test they were asked to 'think aloud', and 
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TAHl K 6.3 .SAMI'LH Rl'CAI.I, AND K VALUATION STKA I 1 <;il S IDl-NTII II l> ItV 
hELOMANN AND ST1:MMI:R (1987) 



Strategy type 



lixatuple 



A. Recall 

By structural analysis 

Kecill by adding letiers/ 
syllables to beginning of 
item 

Recall by substitution 



B. F.i'aluJtioti 

Check on meaniiig of item 



Check on form of item 



S (Itxiking for item in 'jc peiise c] '): je [k-ii 

aj5.nn a relative pronoun st;irrs wiih t) A A A 

S (subject is looking to item 't.isiiJ.i' in 'i l .nri.- Ics 

fast '): el aire ii's fast A A fasro not f.isr.i A 

hni A I really don't know. 

S (subject is lix)kiiig lor diraii; in 'je Jir ' .mil 

uses 'something' as .1 (iller): je tii ilir soiiK-iinng . . 
(contiiuies reading). 

S (ir.nislates se miieren dv calor into uKiihcr 
tongue): ... si estaii querrad.is A A A A s-s c 
nuiercn de calor ihcy die with li A yes .se imieren 
(writes '.sc') dc calor (writes 'de') . . . 

S (subject is checking on endings of rctroiwciu): . . 
pour dies A A representeni has to !<• plural ... or 
is it represent la vie quotidienne A A no has gol tt 
he the scenes. 



their utterances were tape-recorded. The subjects and researchers then rei 
rospectively reviewed the audiotape, and the subject was invited to coniiiieii 
on his or her utterances and Ix'haviour, as well as answer the interviewer 
questions. The researchers then analysed the responses and developed a tyjjol 
ogy of strategies used by the subjects. A strategy was defined as 'a potcntiall 
conscious plan for solving what, to the individual, presents itself as a probleii 
in reaching a particular goal' (Feldmann and Stemmer 1987: 258). Two type 
of strategies were identified: recall strategies and evaluation strategics. Recal 
strategies were employed when the item was not retrieved automatically 
whereas evaluation strategies were used to check on the appropriateness o 
the retrieved item. Table 6.3 sets out a sample of the strategies identified ii 
the research. 

The researchers suggest that the strategies can Ik- ranged on a coiuiiuiun 
from top-down processing strategies (for example, recall of [>ast situation) it 
bortom-up processing strategies (for example, structural analysis), althougl 
they admit that strategies cannot be unambiguously assigned a place on tlu 
continuum. In substantive terms, theredid not seem to be differences in srrat 
egy preferences according to the target language being studied. Finally, tlu 
question of the construct validity of the C-test was left unanswereil. 
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In a fascinating investigation of classroom interaction from the learner's 
perspective, Sliniani (1992) used retrospection to obtain data on what learners 
felt they had learned from particular lessons (what she calls 'uptake'). At the 
end of every lesson she observed, learners were given a questionnaire, or 
'Uptake Recall Chart', on which they were required to recall everything they 
had learned in terms of the following categories: 

1. Grammar 

2. Words and phrases 

3. Spelling 

4. Pronunciation 

5. Punctuation 

6. Ways of using the language 

7. Suggestions about using the language 

8. Other 

Around three hours later, learners were given the uptake recall chart they had 
completed at the end of the lesson as well as a second questionnaire, called an 
'Uptake Identification Probe'. This required the learners to differentiate 
Ix^twcen the items they believed they had really learned in that particular les- 
son and those they bad encountered in a previous lesson. Once uptake items 
had been documented, the researcher went back to the lessons (which were 
audio-recorded) and located the points in the lesson where the events 
occurred. Learners were also interviewed by the researcher. 

Methodologically, this study is of interest because of the care which Sli- 
niani took, through the use of two questionnaires and an interview, to obtain 
reliable and valid data. She makes the point that while the procedures raised 
subjects' consciousness of the learning process and their role within it, the 
data collection procedures may also have had an effect on the learners' behav- 
iour. However, she discounts this po.ssibility on the grounds that the data col- 
lection took place over a relatively short period of time (six weeks). 

In substantive terms, Slimani provides a vivid portrait of the contrast 
Iwtwccn the 'syllabus as reality' in contrast with the 'syllabus as plan' (sec also 
Nunan 1988). I'licrc were, in the classr(M)m data, 256 items which were topi- 
caliscd by the teacher, and therefore made available for the learners to learn. 
Of these, only 112 were claimed by at least one student to have been learned. 
Ninety-two items (almost 34%) went completely unnoticed. Slimani 
concludes: 

I TIicso figures I provide us with a picture of the 'syllabus as reality' as opposed to 
the 'syllabus as ]>lair. The former represents what actually happens in the midst of 
interactive work done by the participants. The on-going interaction leads to the 
creation of a whole range of learning opportunities, some of which are the results of 
the tc-.ulicr's phin; others arise as a by-product of the plan, but some others arise 
iiuicjieiuiently of any intentions, [>erhaps as a by-product of classroom interaction. 
(Sliniani 1992: 209) 
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In this section, we have looked at techniques employing retrospection. I 
pointed out that as the length of time between the task and the retrospection 
increases, so also do threats to the reliability of the research. Despite this, there 
are certain circumstances in which retrospection, rather than introspection, 
is to be preferred. In investigating such things as teachers' on-line decision- 
making and learners' test-taking behaviour, it is simply not possible for the 
subjects to act and reflect/ report simultaneously. 

A sample study using introspection 

In this section, we shall look in detail at a study utilising some of the tech- 
niques discussed in the preceding sections. The study I have chosen is Haas- 
trup's 1987 investigation of learners' lexical inferencing procedures, pub- 
lished in Faerch and Kasper's book on introspection in second language 
research. 

question/hypothesis 

What knowledge sources do learners at different proficiency levels make use 
of, and how do they use the different sources to infer the meaning of 
unknown words? 

significance/ VALUE OF THE STUDY 

An investigation of the inferencing procedures employed by second language 
learners when confronted by an unknown lexical item is of interest for two 
reasons. In the first place, it should illuminate the ways learners form hypoth- 
eses. Secondly, it should facilitate the development of a model of L2 reception. 

SUBJECTS 

Subjects were 124 Danish high school students who were learning English as 
a foreign language. Half of the subjects were judged to have high proficiency 
in English, and half were judged to have low proliciency. Subjects were put 
into pairs so that there were 31 pairs of high proficiency learners and 31 pairs 
of low proficiency learners. 

PROCEDURE 

The dyads were given a simplified passage to read. The passage contained 25 
unknown test words in a comprehensible context, as the following sample 
text shows: 

At the beginning of the nineteenth century some of the Zulu clans were ruled by a 
king called Chaka. He was a clever military leader with insatiable [Xi\\x\ca\ 
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ambitions. He won most of south-eastern Africa and united all the Zulu clans into 
one great empire, rlic Zulu nation. Soon afterwards, however, the downfall of the 
Zulu empiit. started by dissenlion among the blacks themselves, and ended in 
conflicts with the whites, (p. 198) 

The dyads were asked to read the texts and to guess the meaning of the 
unknown words (which were italicized), verbalising all their thoughts as they 
did so. The discussions are transcribed and analysed. 

Half of the pairs also took part in a retrospection exercise. These pairs were 
monitored by two researchers as they completed the think-aloud task. They 
were then interviewed and asked to explain certain asf)ects of the think-aloud 
task, such as why they made lengthy pauses and what led them to suggest 
certain meanings. 

TYPE OF DATA 

The study yielded two types of protocols: the dyadic interactions and the ret- 
rospective statements. A sample think-aloud protocol from the study is repro- 
duced below: 

Test word in context: In the rich world many diseases are caused by affluence. 
Think-aloud protocol: 

A. affluence - do you think it is the opposite of influence? (A -|- B laugh) - non- 
influence. 
B: what does fluence mean.' 

A: I don't know - I know what influence means - fluence (A laughs) 
B: isn't the idea that in the rich world you don't catch diseases - they are not 
infectious - it's more mental problems, perhaps - mental influence more than in 
the poor world when it's simply infection - don't you think.' 
A: well, yes 

B: shall we say external influence - that fits with influence too 
A: yes that's okay 

(Haastrupl987: 198) 

TYPE OF ANALYSIS 

The protocols were analysed qualitatively, and a taxonomy of cue types was 
established. The researcher identified three main types of cue: 

interlingual: cues based on LI (loan words in LI or knowledge of foreign lan- 
guages) other than English 
iniralinguat: cues based on knowledge of English 

contextual: cues based on the text or on the informants' knowledge of the 
world 

Table 6.4 illustrates the ways in which the protocols were analysed. 
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Test word in context: Doctors should analyse why people Ix-come ill rather than 
take such a keen interest in the curative effect of medicine. 



Excerpts from protM ol 




Aniilysis 


Thinking alouil Retrospection 


Cues 


} l\pi>thfses 



A: this is easy f 

think - it 

must be 

something 

with the 

therapeutic 

effect 
B: yes 
A: of the 

medicine 
B: yes it has to do 

with kurere' 



very easy - one 
knows cure 
and then it fits 
the context 



B: a bit of guessing 

- something to 
do with kurere 

- if one were to 
use something 
that resenibles 
it 



intralingnal, lexis 
semantics 



contextual, the 
immediate 
context of the 
test word 
interlingual, 1.1, 
lexis + semantics 



ihera(H:utic 



'Kurere in Danish means "cure" (verb in the infinitive). 
Source: Reprinted with permission from Haastrup (1987: 201). 



CONCLUSIONS 

In substantive terms, the researcher found that there were tlircc main sources 
of information drawn on by the subjects. There were cues based on the first 
language, cues based on current knowledge of the target language, niul cues 
based on knowledge of the world. The differences between the high profi- 
ciency and low proficiency groups were slight. 

Methodologically, Haastrup found problems with lx)th of the data collec- 
tion techniques used. Overall, retrospection was less useful than thinking 
aloud, adding little to the data yielded by the think-aloud technique. Nev- 
ertheless, the two techniques in combination proved superior to either in 
isolation. 

The main problems with the method of thinking aloud arc (I) the varied 
quantity and varied informative value of the data, which makes identification 
of procedures difficult, and in the most unfortunate cases leaves the analyst 
to infer on the basis of the products only (the written result of the infereiicing 
task) and (2) the difficulty in controlling sociopsychological variables for peer 
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tlimkinj; nloiicl. For retrospection, the problems are that the data are only 
partly informant initiated, and that it is difficult for the researcher to strike 
n h.i 1.1 ncc between the too-loose and too-controlled interview. To some extent 
these shortcomings arc overcome when riie two methods are combined. Such 
a procedure provides distinct advantages. Firstly, the two sessions provide a 
larger c]iiantity of data than cither in isolation. Secondly, the quality of the 
ilata are improved in two ways: (I) by using informant-initiated data as the 
starting point and ctiriching them by eliciting additional information, and (2) 
by using pair work that invites the verbali.sation of thought, supplemented by 
deeper probing into the individual's thought processes. By using the methods 
as complements, one has the best of both worlds, although for this particular 
study retrospection did not add a great deal of information (Haastrup 1987: 
211). 

Conclusion 

Ititrospcctive mctluHls have grown in popularity in recent years, as research- 
ers have experimented with different ways of finding out how learners go 
about learning and using language, and also as the questions researchers ask 
become MU)re sophisticated and coinplex. In the next few years the popularity 
of such methods should continue to grow. Diaries, logs, and lournals in par- 
ticular have a great deal to offer research in foreign language learning. These 
can be kept either by teachers or learners or both, and may be used as the main 
data collection t<x)l, or as a supplement to other t(X)ls. 

Questions and tasks 

1. Get three or four inilividnals to complete the NPF.HPA anagram task, 
thinking alouil as they do so. Kecord, transcrilx', and analyse the protocols. 
To what extent do the transcripts lx!nr out the oliservations made by Ericsson 
and Simon? Arc any other strategics apparent in the data? 

2. List the advantages and disadvantages of the dilferent methods di.scussed 
in the chapter, ('an you suggest a question which might be investigated by 
one or more methods? 

}. The journal extracts which follow have lx;en taken from Schmidt and 
Frota (19S.5). Read through them and then complete the tasks which follow. 

Ii>ifni,il entry. Week J 

[ lie first department meeting was tiKlay. ... It started off well, and I actually got 
the gist of the first topic . . . alxnit university personnel policies and changes in the 
rcquircniciirs for ap[)(>iiifiiiciit and promotion to various ranks bu'vcis, my word for 
the tlay). No iloiiht i understood that much because its all xoo familiar in my own 
work, ami the cljair|XTs<>n had a clear diagram on the blacklx)ard. However, after 
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that topic was closed, I did not understand another word for the remainder of the 
meeting, which dragged on to 9 p.m. 1 am totally exhausted. The only observation 1 
can report is that turn-taking rules for meetings in Brazil are different from those in 
the U.S. There was a great deal of overlap among participants, and the general 
atmosphere was a lot rowdier than what I'm used to. (pp. 242-243) 

Journal entry. Week 3 

Just came back from [a sidewalk cafej, where 1 finally had something close to a 
conversation with someone . . . another nonnative speaker ... a linguist, half 
Chinese, half Spanish, who works on comparative Romance phonology somewhere 

in France I was amazed that 1 could understand a fair bit of what he said. He 

spoke slowly and with difficulty and it didn't sound like Portuguese, but words ^ 
only. No Portuguese sentences 1 am sure . . . Now 1 must break out of the trap I'm 
in. 1 want to be fluent in Portuguese by the time 1 leave here, and I want to get 
started. Krashen is absolutely right that native speakers are generally unable or 
unwilling to provide comprehensible input to beginning nonnative speakers. I 
wouldn't waste my time talkmg to someone who knew as little English as I know 
Portuguese. 1 know some people prefer to wait and get settled before tackling a 
language, but in my case this silent period has been maddening and against my will. 
I can't wait for class to start on Monday, (p. 243) 

Journal entry. Week 6 

L and I are still giving each other a hard time. Today in class, K's sentence in a 
substitution drill had a negative before the vert), followed by nada. 1 wanted to find 
out whether other double negatives are possible, so when it was my turn I said en 
nao conbecia niguem ['I didn't know anyone']. This was not the sentence 1 was 
supposed to produce. I don't remember whether L corrected me to algueni or not. I 
only remember her annoyance that I was not performing the drill as I was supposed 
to. so 1 didn't find out what I was after. But in general, the class is OK now, even 
though much too structured. We start out every day with "what did you do 
yesterday?" and always end with "what are you going to do today after class?" In 
between, we have the structure of the day, each beginning with explanations and a 
few examples and far too many drills. I don't mind the drills as much as at first, but 
1 will not say sentences which I don't understand, so I ask a lot of questions. This 
has not endeared me to L, but I think we are now getting to the point of grudgmg 
mutual respect, (p. 244) 

Journal entry. Week? 

S and 1 made the first recording today for our project. I felt very strange at first, 
because this was absolutely the first time we have ever talked to each other in 
Portuguese . . . This is somewhat different from my difficulty in using Portuguese 
with native speakers of English. In that case, it's just so much easier to say it in 
English that it's hard to avoid switching. But with native speakers of Portuguese 
who are English teachers ... I am ashamed to show my ignorance of their language 
when they speak mine so well. With monolingual, it's different: we are equally 
ignorant of each other's language and I feel no embarrassment at all. So S and I 
talked in English right up to the moment she turned the tape-recorder on. Then it 
was fine. It was good to find that I wasn't as embarrassed as 1 expected to be, in tact 
quite comfortable, (pp. 244-245) 
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Journal entry, Week 7 

Last night we [the members of the Portuguese class] and l.'s other class went to a 
Japanese restaurant in Copacabana ... an official exercise in communication 
outside of the class . . . The event went well ... 1 was proud of myself because I 
managed to produce a couple of instances of tao + adj. and ttuis Jo que 
constructions [comparatives], which is what we practiced in class yesterday 
morning. K noticed it, and said she was impressed because she never remembers to 
use in conversation what we learn in class. I think it's easier for me because I've 
been making it a habit, and . . . simply because 1 have fewer resources [than K] to 
draw on. Whatever 1 can get, 1 need. (p. 245) 

journal entry. Week 12 

Portuguese 3 met for the first time this morning. From my p>oint of view, it's a 
disaster. The first sign that things were not going to be perfect was when I arrived 
and found 17-18 people outside a classroom that only holds 10. K was there and 
told me . . . Portuguese 4 had been cancelled and everyone is being put into one 
class. . . . For the group as a whole, the average length of residence in Brazil seemed 
to be about 4 years ... I do not belong in this group. The class started off with a 
discussion of the imperfeito vs. perfeito . . . Then Y said we would do a 
communicative exercise. She got us into groups of three and passed out pictures. 
The exercise was for each of us to make up three sentences about our picture, using 
the present subjunctive. I have no idea how you make the present subjunctive or 
any other subjunctive . . . What really bothered me most was that Y speaks so 
rapidly that I didn't understand much of what she said at all. She apologized for it, 
said that everyone tells her she speaks much faster than most native speakers, but 
she can't control it. I don't know what to do about this situation. I have the option 
of going back to Portuguese 2, but I might be bored there . . . I'm tempted to drop 
completely . - - I'm doing pretty well outside of class, meeting j>eople constantly, 
speaking Portuguese 2-3 hours every day and I think learning something from 
almost every interaction ... 1 have until Monday to change my mind, because 
that's when I have to pay my tuition, (p. 246) 

From your reading of the extracts, what would you say are the social and 
interactional variables which may foster or inhibit language acquisition? Sum- 
marise your thoughts by completing the following table. 





In the classroom 


In naturalistic environments 


Facilitates 






Inhibits 







a. Formulate a research question or hypothesis relating to the effect of social 
and interactional variables on second language acquisition. 

b. Do you find the data from the journal entries convincing, or are they too 

anecdotal to be of any real value? 

c. Study the following statement from one of the researchers. 
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Research in second language acquisition (SLA) has to date typically concentrated on 
the acquisition of systems central to linguistic analysis, ^-sixvi-illy morphology and 
syntax. Indeed, SLA has virtually Ix-cn defined (unplicitly, in most cases) as die 
acquisition of linguistic forms, structures and rules. Only recently has the re Wen 
widespread recognition among .SLA researchers that it is important to study the 
acquisition of other components of language ability as well, esfjecially those 
interactional and stKial aspects of language ability that are frequently referred to 
under the rubric of 'communicative comf>etence'. (Schmidt 1VS,3: 137) 

Based on the journal extracts, how do you think the authors would define 
communicative competence? What are the different components making up 
the construct? 

4. Find a study in the literature which utilises one or more of the techniques 
discussed in this chapter. Summarise and critique the study by supplying the 
following information: 

a. Question/hypothesis 

b. Significance/value of the study 

c. Subjects 

d. Procedure 

e. Type of data 

f. Type of analysis 

g. Conclusions 

h. Further research 

i. Critique 

5. With reference to your own professional context, design a study utilising 
one of the introspective methods outlined in the chapter. 

Further reading 

Without doubt, the most important collection of papers on introspection in 
language research is Faercli and Rasper's Introspection in Second Uingna^e 
Research. It contains a mixture of position papers on the use of introspection 
in both cognitive psychology and applied linguistics, as well as research 
reports. A particularly useful paper in the collection is Grotjahn's, which 
locates introspection within a conceptual research framework. 

For a practical introduction to techniques for collecting data through dia- 
ries, see Bailey (1990). 
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Before I came here I was knowing all the English language tense{s) . . . present tense 
. . . pnst tense . . . presetit (icrfect tense . . . perfect tense . . . future tense . . . future 
in the past . . . everything ... I was knowing ... I am knowing now ... I just 
asked, cr, one day the boss, I said to him "How you knowing this tense?" for 
example 'go' . . . How can you use this word? . . . past tense? present tense? the 
other tense? He just looked me like that ... he told me "I don't know Genghis." 
This is Australian people. 1 am Turkish f)eople. I am knowing, he doesn't know. 
Can you explain this? 

(Genghis, cited in Johnston 1987) 

Studies utilising elicitation are extremely common in the applied linguistics 
literature. In fact, in a recent survey I found that it was the most frequently 
employed data collection method, being used in half of the studies analysed 
(Nunan 1991b). Elicitation techniques have been a feature of second language 
acquisition research since the original morpheme order studies of the 1970s, 
which collected their data through the use of the Bilingual Syntax Measure 
(this device is explained in the next section). Elicitation techniques vary enor- 
nuHisly in scope, aim, and purpose. They include studies which obtain their 
ilata by means of a stimulus, such as a picture, diagram, or standardised test, 
as well as those based on questionnaire, survey, and interview data. 

In this chapter, wc shall only be able to take a selective look at some of the 
more commonly used instruments. If you are interested in exploring any of 
the techniques reviewed here in greater detail, you will find numerous refer- 
ences in the lioiiy of the chapter. This chapter addresses the following 
questions: 

- What is elicitation and how has it been used? 

- How is elicitation used to obtain production data from learners? 

- What are the strengths and weaknesses of surveys, questionnaires, and 
interviews, and how can they be used in applied linguistic research? 

- How can we quantify qualitative data from surveys, questionnaires, and 
interviews? 

Production tasks 

In investigating language learning and use, one can attempt to obtain natu- 
ralistic samples from learners as they interact in the target language. The 

l.V> 
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problem with such a procedure is that it can be extremely time-consuming 
and difficult. In addition, it may not result in the outcomes one desires. The 
learners may simply not use the language items which the researchers are 
interested in. Focused investigations (that is, investigations in which the 
researchers have an idea of the linguistic features they are looking for) 
attempt to overcome these problems by employing elicitation to obtain sam- 
ples of learner language for linguistic analysis. For example, second language 
acquisition studies of the type carried out by Dulay and Burt (1973, 1974) and 
Bailey, JVladden, and Kraslien (1974) were designed to provide the researchers 
with evidence on the appearance of certain grammatical morphemes. The 
aim of the research was to determine the order in which these particular mor- 
phemes appeared, and the effect of such variables as instruction and first lan- 
guage background on the order of acquisition. 

The elicitation device used by these researchers was known as the Bilingual 
Syntax Measure. This consisted of a scries of cartoonlike drawing. Subjects 
were shown the drawings and asked a series of questions which were designed 
to elicit the target language items under investigation. More recently, a sim- 
ilar technique has been used to obtain elicitation data from immigrants. The 
Interview Test of English as a Second Language (ITESL) is designed to obtain 
information from immigrants in Australia for diagnosis, placement, and 
remediation (Griffin 1986). The test consists of twenty items designed to elicit 
twenty target grammatical items. Each item contains a stimulus picture, cue 
questions, instructions for the test administrator, and a set of scoring criteria. 
The following sample illustrates the test items. 

Item 6: Test probe for persona! pronouns 

Stimulus pictures: Picture 1: Illustration of man working in garden 

Picture 2: Illustration of woman working in garden 
Picture 3: Illustration of man and woman working in garden 

Instructions to tester: 

DO SAY 

Point to picture of man working and say: Lxx>k at him. 

(emphasise "he's") He's working. 

Point to picture of woman working and say: And herf 

Point to picture of man and woman working: 

Indicate both people and say: And them? 

Scoring criteria: 

0 neither she nor they is used as required 

1 one of she or they is used 

2 both she or they are used as required 

All responses may include hesitation, uneven fluency, and self-correction. 
Pronunciation can be ptwr but the required pronoun must be clearly understood. 
The pronoun may occur with or without the verb 'to be'. That is she 's, she is, and 
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TABLE 7.1 HYPOTHESISED ORDER OF 
ACQUISITION ACCORDING TO THE 
INTERVIEW TEST OF ENGLISH FOR 
MIGRANTS 


_ ... 

Kjratnnuitical ttctn 




nouns 


1 


verbs 


2 


adjectives 


3 


verb be 


4 


possessive pronouns 


5 


personal pronouns 


6 


adverbs of time 


7 


requests 


8 


simple present 


9 


futures 


10 


wh- questions 


11 


present continuous 


12 


directions 


13 


possessive adjectives 


14 


comparatives 


15 


offers 


16 


simple future 


17 


simple past 


18 


infinitives/gerunds 


19 


1st conditional 


20 



she are all considered acceptable. Similarly they're, they are, and they are all 
considered acceptable. 

(adapted from Griffin 1986) 

The authors of this rest claim that in the course of developing it, they discov- 
ered a developmental order of grammatical acquisition. The test was admin- 
istered to a large sample of subjects whose scores were analysed according to 
a relatively new statistical model, the partial credit model. The order identi- 
fied by the study is set our in Table 7.1. This order is at variance with that 
established by other studies and is, I believe, an artifact of the elicitarion 
instrument, a point I shall address in grearer detail shortly. 

Research such as this based on elicitation techniques differs from natural- 
istic observation in a number of important respects, most particularly in that 
the researcher determines in advance what is to be investigated. Researchers 
need to be aware of two possible threats to the validity of such investigations. 
The first is that by determining in advance what is going to be considered 
relevant, other potentially important phenomena might be overlooked. The 
other danger, and one which needs to be considered when evaluating research 
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utilising such devices, is the extent to which the results obrainotl are an arti- 
fact of the elicitation devices employed (see, e.}^., Nunan 1987 for a discussion 
on the dangers of deriving implications for second language acquisition irom 
standardised test data). One needs to l->e particularly cautious in making 
claims about acquisition orders based on elicited data, as Kills (198.5) h.is 
pointed out. In the case of the ITESL data, which are at variance- with other 
SLA srudies, it seems clear that the so-called order of acquisition i.s tiic crea- 
tion of the elicitation device and the statistical procedures used ro analyse the 
data. 

There is evidence that the so-called developmental order of acquisition 
uncovered by the ITESL is an artifact of the test itself. Consider the following 
response of a subject to Item 6. 

Tester: Look at him, he's working. He's working. 

Subject: Yeah, working in the . . . er . . . park or . . . or . . . govcniineiit park. 
Tester: And her? 

Subject: And same change flower or pur fresh one. (Hmiii)Or watering, I think so. 

Testen Yeah. And them? 

Subject: And rhem working together. 

(Author's data) 

According to the scoring criteria, this subject gets a score of 0 and is tieemcd 
not to have acquired 'personal pronouns'. However, in response to a question 
later in the rest, she does use personal pronouns appropriately. Similarly, in 
the above response, there is evidence that the subject is capable of using the 
present continuous tense. However, when it comes to the prolx; designed to 
test for evidence of the present continuous, she fails to use the required form, 
as is evidenced by her response: 

Subject 's response: 

Tester: What are these people doing? What arc they doing? 

Subject: Have music, coffee, radio. Listen music .md have toa.st and eup of tea. Lr, 
have a sleep. Girlfriends ... er ... is the girl drink coffee. And, er, excuse 
me this is here? Telephc)ne? Oh, worry, looks like worry. Some Hgiit 
together. 

(Author's data) 

Scoring criteria: 

0 no continuous form used or understandable 

1 a numberofcontinuous forms are used but production of the list contains painful 
hesitation and self-correction; little or no fluency in production of list 

2 a clear list of verbs in the continuous form 

The dangers of making claims about acquisition based on a single production 
task is underlined by Eisenstein, Bailey, and Madden (1982), who carried out 
a study into the acquisition of verb tenses. They used two different tasks ro 
elicit data from their subjects. The first of these was a production task similar 
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to tlic Bilingual Syntax Measure, and the second was an imitation task - the 
researcher read aloud sentences containing the target items, and the subjects 
were required to imitate these. Eiscnstcin ct al. concluded that the second task 
provided more accurate information on the subjects' current state of mor- 
phosyntactic development. The major problem, as they saw it, with the pro- 
duction task, was in interpreting the data that resulted from the task. 

It is evident that serious questions must he raised about data from production tasks. 
When a particular structure dix^ not ap[>car, several alternatives are possible: The 
structure may simply not be present in the grammar of the learner, or the learner 
may have some knowledge of the structure but lack the confidence to use it and 
may Ix: exhibiting an avoidance strategy. A third possibility is that the learner 
knows the structure but has not used it as a matter of chance. When a structure is 
used correctly in a form that has high frequency in the language, it could be part of 
an iinanalyzcd chiuik which d<H.'s not reflect the learner's creative use of grammar. 

(p. m) 

Perhaps tlie best way of guarding against threats to the reliability and validity 
of .studies employing production tasks is to do as Eiscnstein et al. did and 
obtain one's data from tnore than one source. While this increases the prac- 
tical problems of carrying out the research, it greatly enhances the internal 
validity of tiie study and enables the researchers to be more confident in the 
claims they make. 



Surveys 

Surveys are widely used for collecting data in most areas of social inquiry, 
from politics to sociology, from education to linguistics. Surveys of commu- 
nity attitudes, opinions, and practices on many subjects, from current voting 
intentions to eating habits, appear in the popular press with monotonous reg- 
iiiarity. According to C^olicn and Manion (1985), surveys are the most com- 
monly used descriptive method in educational research, and may vary in 
.scope from large-scale governmental investigations througii to small-scale 
studies carried out by a single researcher. The purpose of a survey is generally 
to obtain a snapshot of conditions, attitudes, and/or events at a single point 
in time. 

Survey research is distinguished frotn experitnental studies in a number of 
important respects. The most significant difference concerns the role of the 
researcher. As we saw in Cliapter 2, the experimental researcher manipulates 
the environment in order to examine the interaction among variables. In sur- 
vey research, on the other hand, 

ilic researclier tldesn't "do" anything to the t)b|ects or subjects of research, except 
oliserve ihcm or ask them lo provide data. The research consists of collecting data 
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Step 1: Define objectives 

Step 2: Identify target population 

Step 3: Literature review 

Step 4: Determine sample 

Step 5: Identify survey instruments 
Step 6: Design survey procedures 
Step 7: identify analytical procedures 
Step 8: Determine reporting procedure 



Elicitation techniques 

What do we want to find out? 

Who do we want to know about? 

What have others said/discovered 
at)out the issue? 

How many subjects should we 
survey, and how will we identify 
these? 

How will the data be collected: 
questionnaire/interview? 

How will the data collection actually 
be carried out? 

How will the data be assembled 
and analysed? 

How will results be written up and 
presented? 



Figure 7. 1 Steps in carrying out a survey 

on things or people as they are, without trying to alter anything. A survey 
researcher might want to know about teachers' honest attitudes toward their school 
principals, unaltered by the act of asking. The more intrusive a survey, the lower 
the chances that it will accurately reflect real conditions. (Jaeger 1988: 307) 

In carrying out a survey, one works through a series of steps similar to those 
for other types of research. These are set out in Figure 7.1. 

One of the most important questions a survey researcher must confront is: 
What is the population covered by the survey.^ Political surveys, particularly 
those preceding an election, generally purport to cover the entire population 
of eligible voters. Of course it would not be practicable to obtain data from 
the entire population (in fact, this is exactly what the election itself purports 
to do), and a major task for the researcher is to select a representative sample 
from the population as a whole. Perhaps the most comprehensive type of sur- 
vey is the national census, which aims to obtain data on every individual in 
the nation. In more modest investigations it may in fact be possible to survey 
the entire population. For example, a researcher who has been commissioned 
to evaluate the introduction of a computer-assisted language program in a 
single school district may be able to survey all of the language teachers who 
have used the new program. 

In those instances in which it is not feasible to collect data from the entire 
population, the researcher must resort to sampling. The concern here is to 
ensure that the sample is representative of the population as a whole. Numer- 
ous sampling strategies are open to the survey researcher, as Table 7.2 shows. 
(These strategies have been adapted from Cohen and Manion 1985.) The first 
four strategies yield probability samples. Using these procedures, we can 



141 



Research methods in language learning 

TABLE 7.2 STRATKGIES KOR SURVEY SAMl'LINC; 



Strategy Procedure 



1. 


Simple random 


Select subjects at random from a list of the population. 


2. 


Systematic 


Select subjects in a systematic rather than random fashion 
(e.g., select every twentieth person). 


3. 


Stratified 


Subdivide population into subgroups (e.g., male/female) and 
randomly sample from subgroups. 


4. 


Cluster 


Restrict one's selection to a p>articular subgroup from within 
the population (e.g., randomly selecting schools from within 
a particular school district rather than the entire state or 
country). 


5. 


Convenience 


Choose nearest individuals and continue the process until 
the requisite number has been obtained. 


6. 


Purposive 


Subjects are handpicked by the researcher on the basis of 
his/her own estimate of their typicality. 



Source: Adapted from Cohen and Manion (1985). 



determine the probability of selection of each respondent. Procedures 5 and 
6, on the other hand, will result in non-probability samples, in which the prob- 
ability of selection is unknown. Small-scale studies may decide to use non- 
probability samples because they are easier to establish and, in consequence, 
cheaper. They may also be perfectly satisfactory for a preliminary or pilot 
study whose aim is to trial survey instruments and procedures, not to obtain 
data which can be generalised from sample to population. 

The question of how big a sample should be is taken up by Fowler (1988). 
He dismisses the common misconception that the adequacy of a sample 
depends on the fraction of the population included in that sample, arguing 
that 'a sample of 150 people will describea population of 15,000 or 15 million 
with virtually the same degree of accuracy, assuming all other asp>ects of the 
sample design and sampling procedures were the same' (Fowler 1988: 41). A 
key consideration in determining sample size concerns the subgroups within 
a population that may need to be separately identified - for example, whether 
to separate men from women, or whether to separate the responses from 
members of different ethnic groups. 

Survey data are collected through questionnaires or interviews, or a com- 
bination of questionnaire and interview. The construction of questionnaires 
and interview schedules that yield valid and reliable data is much more com- 
plex than might at first be thought. In the next section, we shall look in some 
detail at questionnaire construction and at the analysis of questionnaire data. 
Then we shall turn our attention to interviews. 
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1 have found that among graduate students, the qucsti()nn.iirc is a relatively 
popular means of collecting data. It enables the rescarclier to collect data in 
field settings, and the data themselves are more amenable to qunntlHcation 
than discursive data such as free-form fieldnotes, participant ob-servcrs' jour- 
nals, the transcripts of oral language. However, as I indicated at the conclu- 
sion of the preceding section, the construction of vnlii! and rcliahk' qiustion- 
naircs is a highly specialised business. In this section 1 shall set out some of riic 
pitfalls to be aware of in questionnaire construction. 

Types of questions 

Questionnaire items can be relatively closed or open ended. A closed item is 
one in which the range of possible responses is determined by the researcher, 
for exainple: 'Foreign languages should be compulsory in high schtx)!. Agree/ 
neutral /disagree'. An open item is one in which the subject can Jccidc what 
to say and how to say it, for example: 'What do you think ab<)ut the profx)sal 
that foreign languages should be compulsory in high sch(x)l?' Questionnaires 
can consist entirely of closed questions, entirely of open questions, or a mix- 
ture of closed and open questions. While resixjnses to closetl questions are 
easier to collate and analyse, one often obtains more useful information from 
open questions. It is also likely that responses to open questions will more 
accurately reflect what the respondent wants to say. 

Youngman (1986), cited in Bell (1987), has subcategorised closed questions. 
His categories, along with examples, are set out in Table 7.3. 

Question wording 

When constructing a questionnaire, one needs to pay careful attention to the 
wording of the questions. As we have already seen, a danger with any type 
of elicitation device is that the responses one gets will be artifacts of the elic- 
itation devices themselves. It is particularly important that the researchers not 
reveal their own attitudes through leading questions such as the following: 
*Do you think that the concept of iearner-centredness is Utopian and unreal- 
istic?' Questions should not be complex and confusing, nor should they ask 
more than one thing at a time. The following example from Cohen and Man- 
ion (1985: 105-107) is likely to be confusing to respondents: 

Would you prefer a short, non-award course (3, 4 or 5 sessions) with part-day 
release (e.g. Wednesday afternoons) and one evening per week attendance with 
financial reimbursement for travel or a longer, non-award course (6, 7, or 8 sessions) 



143 



Research methods in Lm^uaf^c learning 

1 Aiii.i; 7.3 c;i.(isi;i) qukstion typi-.s in survkv questionnaires 



Qiii'slittn 
ly/ic 



Examltlc 



list 

Clnttf^ory 
Ranking 



Sc;ilc 



Qiiniitil y/ 
frcc|iiciicy 

(Hi.l 



liulicntc your qiinlitionions by circling any of the following: 
Jiploinn, B.A., M.A., I'll. I). 

ln(Jic;uc your salary range by circling one of the following: 

less tlmn 20,000 

20,(KH)-40,000 

4(),()00-6(),000 

more tlian 60,000 

iiiink the following from 1 to 4 in order of preference. 
'I like to learn licst by studying': 

- with the whole class 

- in small groups 

- in p.nrs 

- im)epen(.lently 

Circle one of the following to indicate your attitude to the 
following statement: 

'I like to learn rhrougii interacting with native s[)cakers'. 
strongly agree, agree, neutral, disagree, strongly disagree 
(Circle one of the following: 

How often did yoii practise English outside class last week? 
0, 1,2. 3, 4, 5, 6, 7, X. 9, 10, more than 10 

1 low many Nl'SH stutlcnts arc there in the following cias,ses.' 





0-5 


.5-10 


10-15 


15+ 


Year 1 
Year 2 
Year ?, 
Year 4 











Soimc: After Younginan (1986), cited in Bcl!(l987). 



with full -day release, or the whole course designed on part-time release without 

evening attciKlancc? 

(For n ciisciission of this i.ssuc sec Briggs 1986; Brislin, Lonncr, and Thorndikc 
197.?.) 

In language education, another danger to avoid is culturally biased ques- 
tions. Numerous authors, including Briggs (1986), Brindlcy (1990), and Bris- 
lin et al. ( 197.S) have pointed out that there is consiiicrable cross-cultural vari- 
ation concerning the type of information which can be sought by a stranger. 
Major liiffcrence.s may cxi.st Ix-twccn the culture of the interviewer and that 
of the respondent, anil tbe.sc differences may affect the responses given (for 
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examples of problems caused by the interviewer's behaviour and language, sec 
'Questions and Tasks' at the end of this chapter, task 3). Factors such as the 
following should be taken into consideration: 

- the willingness of respondents to make critical statements 

- the willingness of respondents to discuss certain personal topics, such as 
age, salary, or opinions on political and social issues 

- the shared values which can be assumed, for example, the concept of free- 
dom of the press 

- the shared attitudes which can be assumed, for example, the commonly 
held belief among many educators in Western countries that classroom 
learning should be a source of enjoyment for the learner. 

Interpreting responses 

I have fouiiil questionnaires to he an attractive means of collecting data by 
graduate students. Unfortunately, the construction of a reliable questionnaire 
which will tell you what you want to know is difficult and extremely time- 
consuming. When constructing questionnaire items, it is important, first of 
all, to he very clear about the objectives of the study, and each itcin should be 
directly referenced against one or more of the research objectives. You should 
also determine in advance how the data to be gathered will be analysed. A 
trap for the inexperienced researcher is to collect the data and then realise 
that the question was asked in a way which cannot be analysed to answer the 
question. Because of these and other pitfalls, it is imperative to pilot any ques- 
tionnaire which is developed. (In fact, I would argue strongly that all research 
should have a piloting phase.) 

Having constructed, piloted, and administered a questionnaire, one is 
faced with the tasks of collating and interpreting the responses. As 1 have 
already indicated, one of the great advantages of closed questions is that they 
yield responses which can readily be quantified and analysed, particularly if 
one has access to computer statistics packages. Free-form responses from open 
questions, although they may result in more useful/insightful data, are much 
more difficult to quantify, although there arc ways of quantifying this qual- 
itative data. 

As an example of the problems of reducing free-form data to manageable 
proportions, consider the following selected extracts from a study I was 
involved in which, among other things, investigated teacher beliefs about the 
nature of language and learning. The question was: 'State three beliefs you 
have alx)Ut language development that determine the way you teach'. There 
were 372 responses. The following samples indicate the type and range of 
rcsf)onses. 

'Children learn by using the language'. 

'All children benefit from immersion of the written print'. 
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'Language is developed through all curriculum areas'. 
'Children should be allowed to learn any new concepts in their native lan- 
guage if possible'. 

'Children learn best when there is a positive encouraging environment'. 

The immediate task was to synthesise fifteen pages of statements in such a 
way as to reveal possible patterns, yet without distorting or misrepresenting 
the data.iOf course, as soon as one places free-form statements into one cat- 
egory or another one runs the risk of distorting the data. The procedure I 
adopted was to conduct a key word analysis, generating categories from the 
statements made by respondents. This resulted in categories such as 'Immer- 
sion' and 'Learning by doing/exp)erientiar. Examples of categories and the 
statements which generated them are as follows: 

IMMERSION 

'Children need to be immersed in all types of writing/reading literature'. 
'All children benefit from immersion of the written print'. 

LEARNING BY DO I NG / E X P E R I ENTI A L 

'Children 's language develops through experiences so in order for the children 
to gain the most out of any given lesson many exp)eriences should be given'. 
'Children learn by using the language'. 

LANGUAGE ACROSS THE CURRICULUM 

'It occurs across the curriculum and therefore should not be seen as a separate 
subject area'. 

'Language is developed through all curriculum areas'. 

GRAMMAR, STRUCTURE, CORRECTNESS 

'A child needs to be aware of basic grammatical structures'. 
'I believe grammar, spelling and reading are the basis for language 
development'. 

ORAL/WRITTEN LANGUAGE RELATIONSHIPS 

'Spoken language should be mastered (ESL) before written'. 
'There is a strong relationship between oral language development and 
expression and the ability to express oneself in writing'. 

CREATION OF RICH, POSITIVE ENVIRONMENT 

'Children need to be an active part of a rich language environment'. 
'Children learn best when there is a positive encouraging environment'. 
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TABLE 7.4 TEACHHRS' BELIEFS ABOUT THE NATURE OE l.ANc;ilAi;K 

ANI1 L EARNINC; 



Category Nunthcr 



Reference to langMge /learning 
Mcxielling 

Immersion 28 

Learning by doing/experiential 25 

Language across the curriculum 22 

Grammar, structures, correctness 20 

Oral/written language relationships H 

Importance of literature LI 

Process approach ^ 

Valuing/impt>rtaiiceof LI ^ 

Direct instruction ^ 

Input ^ 
L I /L2 similarity/contrast 

Integration of four skills 2 

Imitation 2 

Subtotal 202 
Reference to emnromnent /climate 

Creation of rich, positive environment .SO 

Wide variety, many opportunities, frequent practice 21 

Meaningful experiences/context !<> 

Social, collaborative, interactive learning LS 

Subtotal '»2 
Reference to the learner 

Individual differences 22 

Relevance, purposeful 11 

Individual differences, readiness, stages of development 1 1 

Confidence, motivation, experimentation, risk-taking 9 

Home, parental influence ^ 

Active involvement, child centred 5 
Subtotal 

fotal -^72 



Categories were then grouped together according to whether they referred to 
language/learning, the learner, or the climate/environment of learning. This 
enabled the fifteen pages of data to be represented in a table, which is repro- 
duced as Table 7.4. 

Quantifying qualitative data 

The example in the preceding section shows one way in which qualitative 
data can be condensed and quantified. In fact, it is possible to carry out sta- 



147 



Rcscjn h niclbods in Liii^ihif^c learning 

risticai ,)ii;ilyscs on ii;Uii sucli as tlicsc. For example, the researchers might 
want to know whctlier more experienced teachers, or teachers with higher 
degrees, gave ty(X"s of responses whicli were different from less experienced 
or less qualified teachers. They could investigate tliis by sorting out responses 
according to tlie biographical variables of interest (in the following example 
1 have uscil 'experience'), and ilrawing up 2 X 2 contingency tables such as: 





Experienced 
te(i([)crs 
(5+ yejrs' 
experience) 


Inexperienced 
tejchcrs (1-5 
years' 
experience) 


Tntiil 


Rcfcrcncr to l.iiigiingi'/lcarninp 


126 


76 


201 


ReforiMur r<i enviroiiiiK'ni/clinvitc 




67 


102 


Ri k ri-iKc to ibi- le.iriu-r 


19 


49 


68 


loi.il 


ISO 


192 


m 



Using the procciinrc for calculating chi-square set our in Chapter 2, the 
rescarclu-rs can ilctcrnn'ne whether these different responses are significant, 
i'he final, and most difficult, task is to interpret the results. If, in the foregoing 
example, it should transpire that the relatively more experienced teachers 
made significantly n)ore references to principles of language and learning, one 
tiiight be led to hy[Jothesise that as teachers increase their mastery of technical 
aspects of instruction and knowledge of language, their teaching practices 
become less dependent on local eiivirontiicntal and affective factors. Such an 
observ.nion would have implications for teacher education. 

An alternative way of quantifying qualitative data is presented by Lincoln 
arul Guba (1985). (See also tlie description of a 'grounded' approach to qual- 
itative data analysis provided by Strauss 1988.) The first step is to place each 
inilividiial entry onto library index cards. 

1. (iiviii the pile of cards that Ims resulted froiii tlic uiiitiziiiB process, and that will 
Ix' lUdre or less ii.i[)h:i7ardly arrniigcd, select the first card from the pile, read it, 
.irul note its coiiicius. I'liis lirst card represents the first entry in the yct-to-bc- 
n.inuil c.ite[;nry. I'laee it to otie side. 

2. Select the second card, road it, and note its cotucnts. Make a determination on 
tacii or intuitive grounds wliethcr this second card is a "look-alike" or "feel- 
alike" with C'artl I, that is, whether its contents arc "essentially" similar. If so, 
place the second c.ird witii the lirst and proceed to the third card; if not, the 
second cird represents liie first entry in the sccotid yet-to-be-namcd category. 

.V Coiulmie wiili siKcessive cards. For each card decide whether it is a "l(X)k/ 
feel-.ilike ' of cards liiar have already Ikch pl.iced in some provisional category or 
wliedierit represents a new category. Proceed aecorilingly. 

•t. After simie cards iiave U'cn processed the analyst inay feel that a new card 
neither fits any of the prt>visionally established categories nor seems to form a 
new c.itepory. Other cards may now alsolxr recognised as possibly irrelevant to 
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the developing set. These cards should be placed into a miscelLineous pile; they 
should noi be discarded at this point, but should be retained for later review. 

(Lincoln and Guba 1985: .^47-348) 

At the beginning of the process, new categories emerge rapidly, but the rate 
of emergence dech'ncs as more and more cards are processed. Once all the 
cards have been assigned, the analyst must devise categories and labels for 
each of the groups. The internal reliability of the procedure will rest on the 
extent to which an independent analyst agrees on the categories and the 
assignment of individual cards to the different categories. 

Interviews 

The oral interview has been widely used as a research tool in applied linguis- 
tics. In addition to its use in survey research, it has been used by second lan- 
guage acquisition researchers seeking data on stages and proces.ses of acqui- 
sition (Johnston 1985), and also by language testers, who use the oral 
interview as a means of assessing proficiency (Ingram 1984). The 'socioiin- 
guistic interview' has also been used to investigate linguistic variation, con- 
versational analysis (see the preceding chapter), pragmatics, and cross-cul- 
tural communication. 

Interviews can be characterised in terms of their degree of formality, and 
most can be placed on a continuum ranging froin unstructured through .semi- 
structured to structured. An unstructured interview is guided by the 
responses of the interviewee rather than the agenda of the researcher. The 
researcher exercises little or no control, and the direction of the interview is 
relatively unpredictable. In a seini-structured interview, the interviewer has 
a general idea of where he or she wants the interview to go, and what should 
come out of it, but does nor enter the interview with a list of predetermined 
questions. Topics and issues rathet than questions determine the course of the 
interview. In the most formal type, the structured interview, the agenda is 
totally predetermined by the researcher, who works through a list of set ques- 
tions in a predetermined order. The type of interview one chooses will be 
determined by the nature of the research and the degree of control the inter- 
viewer wishes to exert. Because of its flexibility, the semi-structured interview 
has found favour with many researchers, particularly those working within 
an interpretive research tradition. For Dowsctt, the semi-structured interview 

is quite extraordinary - the interactions are incredibly rich and the data indicate 
that you can produce extraordinary evidence about life that you don't get in 
structured interviews or questionnaire methodology - no matter how open ended 
and qualitative you think your questionnaires are attempting to be. It's not the only 
qu.ilitative research technique that will produce rich information about social 
relationships but it docs give you access to social relationships in a quite profound 
way. (Dowsett 19S6: 53) 
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The advantages of the semi-structured interview are, in the first instance, that 
it gives the interviewee a degree of power and control over the course of the 
interview. Secondly, it gives the interviewer a great deal of flexibility. Finally, 
and most profoundly, this form of interview gives one privileged access to 
other people's lives. Dowsett illustrates this third point with a number of mov- 
ing anecdotes. 

I've had a number of interviews with peop\e that have changed my life . . . : 
Interviewing a mother of a girl from one of the schools in the Making the 
Difference study [the study Dowsett was currently researching], who announced, 
about 15 minutes into the interview, that she was dying of cancer and wasn't 
expected to live more than four or five months. She was [xrfectly healthy l(x>king at 
the time. The husband knew but the kids didn't. The family relations, the whole 
education experience of the girl, the way she functioned in her life at that moment, 
was going to be changing quire dramatically in the next four months. As I watched 
that woman talk to me about how they were planning to negotiate the future and 
where it was going and what plans they had for the two girls in ihe family, I 
couldn't be anything but moved. (1986: 55) 

On several occasions in this and other chapters, I have mentioned the bias 
inherent in most research methods. In the case of the oral interview, one 
source of bias is the asymmetrical relationship between the participants. In 
other words, the participants do not have the same rights, and even in an 
unstructured relationship, the interviewer has much more power than the 
interviewee. The inequitable relationship between the interviewer and inter- 
viewee will affect the content of the interview as well as the language which 
is used. In terms of content, biographical factors such as gender and ethnicity 
can affect the validity and reliability of the research. Briggs (1986) cites an 
interview he conducted with an elderly Spanish-speaking couple in New 
Mexico. The interview failed because the interviewees regarded the sessions 
not as interviews, but as 'jjedagogical encounters between two elders and a 
young person with little knowledge of the community' (Briggs 1986: 103). It 
is therefore important for researchers using interviews to incorporate into 
their interpretation of the data the effect of biographical variables, belief sys- 
tems, and so on. 

In linguistic terms, the asymmetry will be reflected in the actual language 
used (van Licr 1989). This is something which will need to be taken into con- 
sideration by second language acquisition researchers who wish to draw con- 
clusions about such things as acquisition orders based on the data. Several 
researchers (see, for example, Johnston 1985) have pointed out that interview- 
ees rarely ask questions, a major problem when one considers that questions 
are important signposts to a learner's stage of morphosyntactic development. 
Various strategies have been tried for overcoming this problem, from role 
plays to the technique of passing the interviewee a note suggesting that they 
ask the interviewer questions. While such strategies may call into question 
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the validity of the research, it is difficult to see how those darn might Ixr 
obtained in any other way. 

Van Lier (1989) provides a uun)lx.'r of alarming examples ol inur.iciions 
between interviewers and second language subjects in which the interviewees 
are treated in a highly questionable manner. In the following extract, for 
example, the interviewer concludes that the subject (a young child) has no 
target language skills. 

I: What's your father's name? 

S: [no response] 

I: What does your father do? 

Where does he work? Where den's your father work? 

Come on girl, talk! talk! Don't Ix- afraid. Where d(Ks your father work? 
S: [no respKJnseJ 

I: What does your mother do? Where is your mother? What dws your iiioilier do? 
S: [no response] 

I: What do you do at home? Do you help your nu)tlier? What lio you ilo? 
S: [ no resfwiisej 

I: ((into the microphone)) Doesn't talk. 

(van I.ier 1989: 5O,)-504) 

There are numerous practical suggestions in the literature for plaiuuiig and 
conducting interviews, ranging from the sensible to the self-evident. Hie fol- 
lowing procedures are recommended by Cohen and Manion (1985), liell 
(1987), and Spradley (1979). 

I. PREPARING THE INTERVIEW SCHEDULE 

Once the research objectives have been established, the researcher has to trans- 
late these into interview questions. Cohen and Manion (1985: .W5) recom- 
mend that the variables under investigation be written dowti by tiainc iti order 
to facilitate this process. At this stage, the question fortnat and response tiioilc 
need to be considered. These will, of course, vary according to the nature of 
the variables under investigation, the type of subjects, the resources available 
to the researcher, and so on. However, regardless of the issues, the researcher 
will still need to decide on the type of questions to be used (open-ended versus 
closed, direct or indirect, etc.) and in what form the responses are to be col- 
lected and analysed. 

2. PILOTING 

Because of the potential problems in the use of the interview that we have 
already identified, it is very important that interview questions are piloted 
with a small sample of subjects before being used. This gives the researcher 
the oppKjrtunity to find out if the questions are yielding the kind of data 
required and to eliminate any questions which may be ambiguous or confus- 
ing to the interviewee. (1 would go further and say that it is important for all 
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clicitation instruments to be thoroughly piloted before being used for 
research.) 

3. SELECTION OF SUBJECTS 

Bell (1987) points out that efforts should be made to secure a representative 
sample, even in a small-scale study. This will involve selecting appropriate 
proportions of subgroups of the population, using whatever variables you 
have determined to be important. In the selection process, it may be necessary 
to negotiate access to subjects or data collection sites with individuals or 
institutions. 

4. ELEMENTS OF THE INTERVIEW 

The Structure of the interview itself will depend on the extent to which the 
sequence of questions is fixed. However, it is possible to delineate a number 
of key dements which may be present. These are as follows (see Cohen and 
Manion 1985: 306 ff and Spr.idlcy 1979: 59-60): 

Briefing ami cx{}landtiau. Before the interview begins, the researcher 
explains the nature of the research and the purpose of the interview to the 
interviewee and answers any questions that he or she may have. This includes 
telling the interviewee how the data are to be used. If the data are to be 
recc)rdcd and, in particular, if the data are to be made available to other peo- 
ple, the interviewee's permission must be sought. 

Questioning. In tiie less structured ethnographic interview a range of ques- 
tion types may he used. In order to encourage the respondent to recount his 
or her experiences, opinions, and so on, the researcher may also use a variety 
of other strategies. These are outlined by Spradley (1979: 67). 

Walker (1 985) makes a number of practical suggestions about the actual con- 
duct of the interview. He considers first of all the physical positioning of the 
interviewer and interviewee, suggesting that sitting side-by-sidc can often 
result in a more productive interview than sitting face-to-face (sitting side-by- 
side can convey the message that the interaction is meant to be cooperative 
rather tlian confrontational). The researcher must also decide how the inter- 
view is going to he recorded. While tape-recording is the obvious choice for 
someone collecting linguistic data, it is not the only option. The other option 
is for the interviewer to take notes and to use these to reconstruct the inter- 
view at a later date. Walker points out that tape-recording and note-taking 
are not simply alternative data collection techniques, but represent quite dif- 
ferent ways of going alx)Ut doing research. Some of the relative strengths and 
weaknesses of the two recording devices arc set out in Table 7.5. 

Aijplicd linguistics researchers arc generally interested in the language used 
by their subjects, even when they are investigating intercultural and attitu- 
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TABLE 7.5 STRENGTHS AND WEAKNESSES OH TAPE-RECORDING AND NOTE- 
TAKING 



Instrument 


Strengths 


Weaknesses 


Tajie- 


Preserves actual language 


Possibility of data overload 


recording 


Naturalistic 


Time-consuming to transcribe 




Objective record 


Context not recorded 




Interviewer's contributions 


Presence of machine offputting 




recorded 


Core issues masked by irrelevancies 




Data can be reanalyscd after 




the event 




Note- 


Central issues/facts recorded 


Recorder bias 


taking 


Context can be recorded 


Actual linguistic data not recorded 




F.conomical 


Encoding may interfere with 




Off-record statements not 


interview 




recorded 


Status of data may be questioned 



dinal phenomena rather than linguistic development. For this reason, they arc 
more likely than other types of social science researchers to want to use tajx; 
recorders. The most sensible procedure would probably be to tape-record 
interviews, but supplement these with written notes. I have found in class- 
room research that taking copious notes in addition to recording the lessons 
under investigation greatly enhances the subsequent tasks of transcribing and 
interpreting the taped lessons. The same should also hold for interviews. 



A sample study 

In this section, we shall look in detail at a research project which used an 
elicitation device for obtaining its data. The study, by Cohen and Olshtain 
(1981), investigated the feasibility of using rating scales to assess sociocultural 
competence. 

AREA FOR INVESTIGATION 

The sociocultural competence of second language speakers. 
RESEARCH QUESTIONS 

Can a rating scale be devclofx-d for asessing sociocultural competence? Do 
non-native speakers misuse or omit semantic formulas in an apology in a way 
that natives would not? Can we also rate utterances as to their stylistic appro- 
priateness in the given context? 
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RATIONALE 

At the time this study was carried out, language researchers were beginning 
to broaden the focus of their attention from the development of grammatical 
competence to other areas of target language development, such as discourse 
and pragmatic competence. The focus of this study was the development of 
soci(Kultural competence, defined as 'the ability to use the appropriate socio- 
cultural rules of speaking (also referred to as "sociolinguistic" rules), i.e., the 
ability to react in a culturally acceptable way in that context and to choose 
stylistically appropriate forms for the context' (p. 113). 

In order to investigate this aspect of language use, however, it is necessary 
to operationalise the construct through an instrument which enables it to be 
measured. It was this concern for a measure of sociocultural competence 
which motivated the study in the first place. Because there are so many speech 
acts constituting sociolinguistic competence, the researchers restricted them- 
selves to just one of these - that of apology. They chose this particular speech 
act because they believed it to be 'emotionally charged', and therefore more 
likely to elicit a natural response than a more emotionally neutral speech act. 

SUBJECTS 

Subjects were 44 college students in Israel. Thirty-two of these were native 
speakers of Hebrew who were enrolled in an EFL program, and who were 
judged to beat an intermediate-high level of proficiency in English. The other 
12 subjects were native speakers of English. The subjects were divided into 
three groups. The native speakers acted as informants for native speakers' 
apologies, 12 of the Hebrew speakers served as informants for apologies in 
comparable situations in Hebrew, and the remaining 20 Hebrew speakers 
served as subjects apologising in English. 

RESEARCH METHOD 

The researchers used a role play based on an elicitation instrument which was 
administered to all subjects. The instrument consisted of the following 
instructions, followed by eight role cards (the 12 Hebrew informants had 
cards which had been translated into Hebrew). 

Instructions 

You will be asked to read eight brief situations calling for an apology. In each case, 
the person whom you owe the apology will speak first. I m\\ role play this person. 
Respond as much as possible as you would in an actual situation. Your response 
will be tape-recorded. Indicate when you've finished reading. 

Sample situations 

You're at a meeting and you say something that one of the participants interprets 
as a personal insult to him. 
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He: "I feel that your last remark wa.v directed ;u im- and I t;ikf oHciim'." 
You: 

You completely forget a crucial meeting at the office wiih your Ixiss. An hour Lui-r 
you call him to apologize. I'he problem is that this is the second time you've 
forgotten such a meeting. Your bos.s gets on the phone and asks you: 
Boss: "What happened to you.'" 
You: 

(Cohen and Olshtain I9S1: \M) 

TYPE OF DATA 

Administration of the instrument resulted in prottKols in bi>th English and 
Hebrew. 

TYPE OF ANALYSIS 

In assessing non-native deviation from native patterns, the rese.irchers used 
the following list of semantic formulas developed by Eraser ( 1 979). While the 
examples given are in English, the formulas must, presumably, Ix- relevant to 
Hebrew as well. 

1. An expression of apology 

a. An expression of regret (e.g., 'I'm sorry') 

b. An offer of apology (e.g., 'I apologise') 

c. A request for forgiveness (e.g., 'Excuse me' or 'Forgive me') 

d. An expression of an excuse (not an overt apology but an excuse which 
serves as an apology) 

2. An acknowledgment of responsibility 

3. An offer of repair 

4. A promise of forbearance (i.e., that won't happen again) 

In analysing the data, the researchers were careful to differentiate between 
deviations from native speaking norms resulting from negative transfer from 
the native language and those resulting from lack of proficiency in the target 
language. 

RESULTS 

1. In certain situations, both the second language subjects and the Hebrew 
informants used semantic formulas considerably less often than native 
English speakers. For example, Hebrew speakers were less likely to express 
an apology than native English speakers in 'insulting someone in a meeting' 
and 'forgetting to take their son shopping'. This would indicate that the devi- 
ations are a result of negative transfer from the target language. 
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2. \i\ sonic siriiatioiis tlic deviation seemed to be a matter of lack of profi- 
ciency rather tlian negative transfer. 

^. In sonic. sitnarions non-native English speakers responded more like native 
English speakers than like Hebrew speakers. 

4. On the question of whether a rating scale could be developed for assessing 
sociocultural competence, tiic researchers felt that they had produced at best 
a crude measure of such competence. 

CKITIQlli; 

This is a carefully thought out and well conducted investigation of an area 
of language development which has been generally neglected. As such, it rep- 
resents an important addition to the applied linguistics literature. However, 
it is always [jossible to identify problems and shortcomings. 

As with any study employing elicitation instruments, there is always the con- 
cern that the results have been determined, at least in part, by the instrument 
itself. In the case of the study we have looked at here, would the subjects have 
responded in these ways if the situations were genuine rather than elicited? 
(I'rohlenis associatcti with the use of elicited data are acknowledged and dis- 
cussetl in detail by the researchers.) The validity of role-play data has been 
extremely contentious amotig those scholars concerned with this kind of 
pragmatic research. However, the question remains as to how one would col- 
lect relevant data on this aspect of pragmatic interaction if the role play were 
not used. 

In addition, the researchers' judgments were based on an examination of 
the percentage of responses assigned to each category. No tests were applied 
to determine whether or not the differences were .statistically significant. 



Conclusion 

In this chapter, we h.ive looked at a range of research instruments which have 
been drawn together under the rubric of 'elicitation*: stimulus pictures and 
role plays, as well as surveys, questionnaires, and interviews. The breadth of 
the chapter reflects the fact that elicitation is probably the single most fre- 
quently used method in language research. 

In terms of intervention and control, elicitation resides somewhere 
between the formal experiment and naturalistic ol^servation. While most 
researchers are aware of threats to the validity of their research posed by the 
use of elicitation devices, in many instances, these devices arc the only prac- 
fic.d mcnis whereby relevant tiata can he collected. 
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Questions and tasks 



1. What are the possible advantages and disadvantages of the different data 
collection procedures set out in the chapter? What are the situations in which 
one might select one strategy rather than another? In terms of external valid- 
ity, which method do you think would be most likely/least likely to give 
results which accurately reflect the population from which the sample is 
drawn? 

2. You have been commissioned by your institution to conduct a survey of 
people who enrol in your courses. The institution is interested in finding out 
why people do the courses and if there is a typical student profile. Part of the 
survey will therefore involve an investigation of their attitudes and motiva- 
tion. Identify some research questions or hypotheses that you might start 
with. 

3. Study the following extracts from van Lier (1989). What 'procedural prob- 
lems' can you identify? How do you think the interviewer's behaviour or lan- 
guage might have influenced the data collected? 



a. (From an oral proficiency interview) 

Interviewer: Where is your mother? What docs your mother do? 

Subject; She's dead. 
Interviewer: Ah - she's dead. Very good. 

(van Lier 1989:499) 



b. (From an interview aimed at eliciting data on the 
development) 
Interviewer 
1. In Colombia do the lobsters have claws? 
3. Claws. LX) they have ... the lobsters, do 
they have have claws (form my hands into 
claws)? 

J. No. The lobsters. Do the lobsters have 
hands? 

7. I don't know how to say it. I know ... 1 
am a lobster. This is my ... 1 am a lobster. 
This is my claw (hands formed like claws). 
9. Do lobsters in Colombia have claws? Like 

this, you know? They pinch people. 
1 1. On Sunday do you catch many lobsters? 
13. Yeah, do you get many? 
15. Oh, do you want to go get your 
dictionary? 

17. How many, how many do you get? How 
many do you catch? 



subject's syntactic 

Subject 
1. Claws? 
4. Octopus? 



6. Huh? 
8. Hm. Hm. 

10. Lobster? 

12. Eh, huh? 

14. F.h. Dictionary? 

16. No. No necessary. 

18. Catch? 
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Interviettfer 

19. On Sunday. Yeah. How many do you 

catch on Sunday? 
21. Thirty or forty.' 
23. What do you do with them.' 
25. The lobsters. What do you do with the 

lobsters? 
27. Yes, where do they go.' 



Subject 

20. Thirty or forty. 

22. Depend. 
24. Huh? 
26. You do? 



28. In morning. 



(Butterworth, cited in Hatch, Wagner Gough, and Peck 1985: 55) 



(From a second language acquisition research 
Observer 

1. Do you know what a question is? 

3. Yeah. What do we mean when we say 

questions? 

5. What do we mean by question marki' 
What do we mean? What does question 
mark mean? 

7. Yeah. But why do we put that mark 
there? 

9. Why do we put that mark there? See, we 
don't have it there. We just have a period 
there. 

10. Do you wrestle? 

12. No? 

14. You like to watch. 
16. You don't like to do it yourself. You'd 
rather watch. Let's say "watch." 
Say "watch." Same thing. Look - watch, 
look - watch. 

Watch. You like to watch 

What are you gonna do tonight? 
You don't know yet? D<) you work at 
home, do the dishes or sweep the floor? 
You water? 
Flowers. 

29. Oh, you wash the mud down and all that. 
What else do you do at home? 



18. 

20. 
21. 
23. 

25. 
27. 



project interview) 
Adolescent 
2. Question mark. 
4. Question? 

6. This a (points) 



8. This. 



11. No. 
13. I like look. 
15. No me wrestle. 
17. Hm? 

19. Watch 



22. Tonight? I don't know. 
24. Water (garbled). 

26. Flowers. 
28. Mud. 
30. Home? 

(Hatch et al. 1985: 54) 



Further reading 

Briggs (1986) provides a wealth of practical information on what to do (and 
what not to do) when interviewing subjects. 

For a general introduction to surveys, questionnaires, and interviews in 
education, a useful text is Cohen and Manion (1985). 
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'When I use a word', Humpty Dumpty s.iid, in rather a scornful tone, 'it me.iiis just 
what I choose it to mean - neither more nor less'. 

(Lewis Carroll, Through the Looking GLiss) 

In this chapter, we shall look at the analysis of interactions iKcurring in luit- 
uralistic environments, that is, in non-experimental, non-elicited environ- 
ments. In doing so, we shall test Humpty Dumpty's assertion tliat speakers 
are free to make words mean whatever they want tlieni to mean. 1 sli.ill devote 
the first section of this chapter to studies of first langiia(;e interaction, focus- 
ing in particular on some of the work which lias been carried out in ciiild- 
adult interaction. This work is of interest to second and foreign language 
researchers because it is frequently claimed that second language ()edagogy 
should be guided by the conditions under which first languages are acquired. 
We shall also look at the procedures developed by interaction analysts for 
studying adult interaction in social and transactional encounters. T lie chap- 
ter addresses the following questions: 

- What is interaction analysis? 

- How can data from children acquiring their first language be analysed? 

- What does the research literature have to say about the relationship 
between first and second language acquisition? 

- What methods exist for the analysis of transactional and interpersonal 
interactions between adults? 



Comparing discourse analysis, interaction analysis, and 
conversation analysis 

Before proceeding with the main business of the chapter, 1 should like to 
address the distinction between discourse analysis, interaction analysis, and 
conversation analysis. In looking at the literature, it quickly becomes appar- 
ent that the distinction is by no means clear-cur, and in many instances, the 
distinction is one of emphasis rather than distinct categories. 1 have found it 
useful to distinguish between the different analytical methods in terms of four 
different factors: firstly, the means whereby the data have been collected; sec- 
ondly, the mode of language which is admitted into the analysis; thirdly. 
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whether the researclier brings to the analysis a predetermined set of analytical 
categories; ;ind finally, whether the focus of attention is essentially linguistic 
or non-linguistic. 

hi looking at the means whereby the data were collected, the salient ques- 
tion is whether the data were collected through some form of elicitation or 
whether the researcher has tried to obtain naturalistic sainplcs for analysis, hi 
general, in all forms of analysis, the preference is for naturally occurring lan- 
guage. However, while conversation analysts rigidly eschew the use of elic- 
ited or invented samples of language, both of these practices are acceptable to 
some discourse analysts. When considering studies based on natural samples 
of language, we need to keep Labov's observer's paradox in mind, and ask 
whether it is ever possible to obtain natural samples of language. In other 
words, is it ever possible, through observation, to collect the sort of language 
.speakers u.se when they are not being observed.' Interaction analysts do not 
use invented .samples of language, and, like conversation analysts, generally 
favour the collection of naturally occurring, non-elicited language. 

The second factor is the mode of language which is admitted into the anal- 
ysis. Discourse analysis is carried out on both written and spoken language 
(although individual analy.sts generally favour one form rather than another). 
Conversation and interaction analysis, on the other hand, arc both concerned 
exclusively with spoken language. 

The third question concerns whether or not the analyst brings to the ana- 
lytical task a predetermined set of categories. Once again, discourse analysis 
is distinguished from conversation and interaction analysis in the use of pre- 
determined analytical categories. Conversation and interaction analysts 
favour a discursive, interpretive type of analysis. 

The final distinction 1 should like to draw here concerns the substantive 
focus :)f analysis. Discourse analysis has developed within linguistics, and it 
is therefore hardly surprising that the analysis is generally carried out in lin- 
guistic terms. In the same way as sentence grammarians are concerned with 
what makes a well formed sentence or utterance, the discourse analyst inves- 
tigates what it is that makes for well-formed discourse. Discourse analysts 
have studied textual factors such as the use of reference and conjunction, 
which contribute to cohesive discourse, as well as the ways in which speech 
acts such as 'inviting', 'apologising' and 'denying' are performed and inter- 
preted within coherent discourse {see, for example. Brown and Yule 1983). 
Conversation analysis, on the other hand, has emerged from a schcx>l of soci- 
ology known as ethnomethodology, and the rules and procedures employed 
are sociological rather than linguistic in character. Substantively, this form of 
analysis investigates such things as the management of turn-taking, repair 
strategics, the resolution of ambiguity, speaker selection, and topical rele- 
vance. It also investigates the way certain speech acts, such as question- 
answer and offer-decline, combine as adjacency pairs; in this regard, conver- 
sation analysis overlaps with discourse analysis. The essential difference is 
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TABLE 8.1 DISTINGUISHING CHARACTERISTICS OF DISCOURSH, CONVERSATION, 
AND INTERACTION ANALYSIS 





Discourse 
analysis 


Conversation 
analysis 


Interaction 
analysis 


Method of generating 


Invented 


Naturalistic 


Elicited 


data 


Elicited 




Naturalistic 




Naturalistic 






Mode 


Spjokcn 


Spoken 


Spwken 




Written 






Type of an.ily.sis 


Categorical 


Interpretive 


Interpretive 


Units of analysis 


Linguistic 


Non-linguistic 


Both linguistic and 








non-linguistic 



that the discourse analyst is concerned with the rhetorical routines realised 

in speech, while the conversation analyst is interested in the social routines 
(for an extended discussion with examples, see Aston 1986). Interaction ana- 
lysts are concerned with both the linguistic and non-linguistic aspects of sp>o- 
ken language, and attempt to articulate links between the linguistically 
focused rhetorical routines and social asjiects of interaction. 

The various features and characteristics which I have discussed are set out 
in summary form in Table 8.1, and this should make clear points of similarity 
and divergence between the three approaches. 

Which of these is the preferred type of analysis? In his book on pragmatics, 
Lcvinson (1983) synthesises the debate in the following manner. 

DA [discourse analysis] theorists can accuse CA [conversation analysts] of being 
inexplicit, or worse, plain nuicldled, alx)ut the theories and conceptual categories 

they arc actually employing in analysis CA practitioners can retort that DA 

theorists arc so busy with premature formalization that they pay scant attention to 
the nature of the data. The main strength of the DA approach is that it promises to 
integrate linguistic findings about intrascntential organization with discourse 
structure; while the strength of the CA p>osition is that the procedures employed 
have already proved them.selves capable of yielding by far the most substantial 
insights that have yet been gained into the organization of conversation. (Levinson 
1983: 287) 

From the foregoing discussion, you can see that interaction analysis shares 
characteristics both with discourse and conversation analysis. It is narrower 
than discourse analysis in tertns of mode and method of generating data, but 
somewhat broader when it comes to analysis. While it shares the same broad, 
interpretive approach to the analysis of data as conversation analysis, it is 
broader in terms of the range of features subjected to analysis. I have chosen 
the term interaction analysis as the title for this chapter because of its breadth 
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of focus. This will allow me, in parts of the chapter, to stray into the domains 
of discourse and conversation analysis. For an alternative approach, which 
subsumes interaction and conversation analysis under discourse analysis, see 
Hatch (1992). 



Child-adult interaction 

As intimated at the beginning of this chapter, it is useful for foreign language 
educators to be aware of research issues and methods in the field of first lan- 
guage acquisition, because numerous methodologists and materials develop- 
ers claim a privileged place for their approach on the basis of a supposed rela- 
tionship to the conditions which account for first language acquisition. Asher 
(1977), for example, claims that his methodology (Total Physical Response) 
is underpinned by important characteristics of first language acquisition. 
These include the comprehension-first principle (that infants have a silent 
period in which they comprehend but do not speak), the here-and-now prin- 
ciple (that parents or primary caregivers only talk to the infant about things 
that are present in the immediate environment), and the physical response 
principle (which is based on the conviction that much early language is in the 
form of imperatives to the child to carry out actions, such as 'pick up the ball', 
'give the blocks to Nana*). By familiarizing ourselves with some of the liter- 
ature on first language acquisition, we shall be better placed to evaluate the 
claims made by these methodologists and materials developers. The research 
is also interesting because of the methods employed for data collection and 
analysis. 

Other claims (and I emphasise the fact that these are claims, not truths) in 
the literature which have been taken up by second language specialists include 
the following: 

1. Contrary to popular belief, the child does not first learn structures which 
are then deployed in conversational interaction. Rather the child learns 
how to 'do' discourse, by interacting verbally with primary caregivers, and 
it is out of this discoursal interaction that syntactic structures develop. 

2. Children learning their first language plan conversations and gamelike 
interactions. Adult interlocutors facilitate this process by providing con- 
versational 'scaffolds' for the child. The adult does a great deal of inter- 
pretive work to ensure that the interaction does not break down. Typi- 
cally, the child is allowed to take the lead, with the adult following and 
supporting. 

3. In child-adult interactions, the focus is on the meanings being conveyed, 
not the forms in which the meanings are couched. Virtually all parental 
corrections relate to violations of meaning, not form. 
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(For detailed discussion on these and other points, see Asher 1977; Hatch 
1978; Krashen 1981, 1982; van Lier 1988; Wells 1981.) 

Not surprisingly, a great deal of research is devoted to recording and ana- 
lysing the developmental patterns and stages through which children pass in 
acquiring the phonological, morphosyntactic, semantic, .inJ pr.igiiiatic sub- 
systems of the target language. Much of this research takes the form of 
detailed case studies. Fletcher (1985), for example, provides a Ixxik length 
study of the language development of one child, Sophie, from the age of 2 
years and 4 months through to 3 years and 11 months. 

In this section, I shall restrict my focus to investigations of children's inter- 
acfional abilities. This work is interesting Ix-cause it is now wiilely believeil 
that children learn interactional skills first and then begin to m;ister other 
aspects of the language, such as the phonology and the granunar of tlie lan- 
guage. (This is the reverse of many second language methods.) The following 
two extracts from Wells (198 1 ) illustrate this point. They arelietween a child, 
Mark, and his mother. As you read the extracts, consider the following 
questions: 

What differences are there between the two extracts? 

Is there any evidence that Mark's language has developed between the first 

and second conversations? (In Extract 1, Mark is 23 months of age, and in 

Extract 2 he is 28 months old.) 
What is the precise nature of Mark's contributions to the interactions? 
What has Mark learned about 'doing' conversations? 
What does his mother do to make the conversation work? 

Extract 1 

Mark is Uxsking in a mirror and sees reflectionsof him.self and his mother. 



1 


Mark: 


Mummy (v) 


2 




Mummy 


3 


Mother: 


What? 


4 


Mark: 


There - there Mark 


5 


Mother: 


Is that Mark? 


6 


Mark: 


Mummy 


7 


Mother: 


Mm 


8 


Mark: 


Mummy 


9 


Mother: 


Yes that's Mummy 


10 


Mark: 


* 


11 




Mummy 


12 




Mummy (v) 


13 


Mother: 


Mm 


14 


Mark: 


There Mummy 


15 




Mummy (v) 


16 




There. Mark there. 


17 


Mother: 


Look at Helen 


18 




She's going to sleep (long pause) 
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Miirk ain sec birds in the garden 

19 Miirk: | | ( = look at that) 

20 Birils Miimniy (v) 

21 Mother: Mm 

22 Mnrk: julw (birds) 

2.^ Mother: What arc they doing? 

24 Mark: Jiibs bread 

25 Mother Oh look 

26 They're caring the berries aren't they? 

27 Mark: Yeh 

28 Mother: That's their food 

29 I liey liavelx-rrics for dinner 
.10 Mark: Oh 

(Wells 1981: 102) 

Ext nu t 2 

Mark has seen a man working in bis garden 



1 Mark: Where man gone? 

2 Where man gone? 
?t Mother: I don't know 

4 1 expeet he's gone inside because it's snowing 

5 Mark: Where man gone? 

6 Motiier: In tiie house 

7 Mark: Uh? 

8 Mother: Into his house 

9 Mark: No 

10 • No 

1 1 Gone to shop Mummy 



i'iie local shop is close to Mark's house 

12 Mother: Gone where? 

1.1 Mark: Gone shop 

14 Mother: To the shop? 

LS Mark: Yeh 

16 Mother; Wh:u's he going to buy? 

17 Mark: F.r - biscuits 

18 Mother: liiseuits nun 

19 Mark: lib? 

20 Mm 

21 Mother: What else? 

22 Mark: F.r - meat 
2.1 Mother: Mm 

24 Mark: Meat 

2.5 F.r - sweeties 

26 Buy a big bag 

27 Mother: lUiy sweet.s? 

28 Mark: Yeh 

29 M - er - man - buy the man buy sweets 
.10 Mother: Will he? 
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31 Mark: Yeh 

32 Daddy buy sweets 

33 Daddy buy sweets 

34 Mother; Why? 

35 Mark: Oh er - [ ] shop 

36 Mark: Mark do buy some -sweets - sweeties 

37 Mark buy some - urn - 

38 Mark buy some - urn - 

39 I did 

(Wells 1981: 107) 

In commenting on Extract 1, Wells argues that despite Mark's obvious 
lack of linguistic resources, in interactional terms, the conversation is remark- 
ably similar to ordinary conversation. Mark is able to take turns appropri- 
ateiy,and is also capable of maintaining a particular conversational topic over 
several turns. In functional terms, Mark's language is pragmatic rather than 
prepositional. For example, in lines 1, 12, and 19 he attempts to initiate an 
interaction; he draws attention to what is currently of interest in lines 4, 6, 
14, and 20, and in lines 27 and 30 he acknowledges his mother's contribu- 
tions. The mother, on the other hand, does a lot of interpretive work to ensure 
that the conversation flows smoothly. She uses simple utterances and restricts 
the semantic content to immediate topics. Pragmatically, she lets Mark take 
the initiative. For example, in lines 5, 9, 18, 23, 25-26, and 28-29 she u.scs 
cohesion to maintain the coherence of the interaction. 'Mark is being given 
the opportunity to learn both the referential links between words and struc- 
tures and the objects and events to which they refer, and also the intrnlin- 
guistic meaning relations that hold between these words which all belong to 
the same semantic field' (Wells 1981: 104). 

In Extract 2, there are numerous indications that Mark's language has pro- 
gressed. In lines 1, 11, 26, and 36 we see an increase in the length and com- 
plexity of his utterances. There is evidence of his ability to respond appropri- 
ately to information-seeking questions in lines 17 and 22. He also refers to 
persons and events which are not present in the here and now. The extract 
also shows the mother continuing to provide a scaffold for the interaction. 
She follows Mark's lead despite his rejection of her suggestion that the man 
has gone inside because of the weather. She checks her understanding through 
clarification requests and confirmation checks in lines 12 and 14, and then 
assists Mark in building a fictional account involving a sweets shop. In fact 
the interaction owes as much to the structure provided by the mother as to 
Mark's contributions: 

The meaning is 'negotiated' and the resulting story is a collaborative achievement. 
Firstly, there is the need for the adult to interpret the child's contribution in the 
light of the immediate context and the focus of joint attention; secondly, to 
maximise uptake, the adult's own contributions need to be closely related to the 
child's preceding communication and current interest; thirdly, whilst being 
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mcxlified in timing, form and content to the child's receptive capacities, these adult 
contributions must also provide the means v^ hercby the child can enlarge his 
linguistic resources and, through them, his understanding of the content of the 
communication (Wells 1981: 108) 

This textual commentary is typical of the analysis engaged in by those who 
favour an interpretive approach to their data. As we saw in Chapter 3, the 
'intensive immersion' in recorded data (Edelsky 1981) is typical of ethnog- 
raphy. The key characteristic of this approach is the suspension of preformed 
judgments and beliefs. It has been suggested that the researcher should 
approach the data in the same way as an ethnographer might investigate an 
unknown culture - not by sp)ecifying in advance what will count as signifi- 
cant, but by assuming that anything that occurs in the data might potentially 
be of significance (van Lier 1988). In this 'intensive immersion in the data', 
Wells shares common ground with the conversation analyst. However, by 
admitting into his analysis an investigation of the development of different 
speech acts, he also draws on techniques and strategies from discourse anal- 
ysis. Wells's research is summarised in Table 8.2. 

Another imjxjrtant issue concerns the context and environment of the 
research. Most forms of interaction analysis take into account the context in 
which the data were collected. In child language studies, contextual factors 
include not only where the interaction takes place and what is happening at 
the time, but also the relationship between the interlocutors. This is exem- 
plified in Extracts 3 and 4, which are from my own data. Both extracts 
involve a child, Rebecca, interacting in two different contexts, with different 
interlocutors. In Extract 3 she is interacting with her older sister and older 
cousins. In Extract 4, on the other hand, she is interacting with her mother. 

As you read these extracts, consider the different communicative roles 
played by Rebecca and the different 'messages' she gets about interaction in 
the two different encounters. Both interactions were recorded on successive 
days when Rebecca was 27 months of age (almost the same age as Mark). 

Extract 3 

[This interaction takes place in Rebecca's bedroom. The older children, Jenny, Zoe, 

and Jodie, are playing 'mums and dads' and want to exclude Rebecca.J 

R: A, a play, a ready ... a mum a dad. 

Je: Darling, you're only a darling, now stay here, stay here. 

Z: 1 want a drink. 

Je: Now darling, it's time to go to ni nis. Lie down, (forces Rebecca down in her 
cot.) Go ni nis. You can stay up darling tonight, (inaud) . . . go ni nis. (puts up 
side of cot) 

R: Ah, ah, darling. How, a shutting you. 

Je: There darling. No. Do you wanna get out.' (Yeah). 

Do you wanna play with your sister? (secured side of cot) 

R: Ah ni nis, ah ni nis. Ni nis. 
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TABLH 8.Z SUMMARY OF FIRST LANGll ACiU ACQUISITION STUDY BY Wl l 1 S (19SI) 

Question How is it that the human infant is able to aa|uire its liist 

language rapidly and successfully? 

Rationale Although we know a considerable amount .ibt)ut the 

organisation of forn^al systems of language in isol.ition Ironi 
the contexts in which they are used, we arc siill l.irgcly 
ignorant about the principles that uiuicrlie the ortlerliness ol 
conversation. We are also ignorant alxnit how the lunnaii 
infant is able to acquire language. 

Subjects A male infant, Mark, followed (ram the ages of 2,5 to 2H 

months, and his mother. 

Procedure Natural samples of language are collected from Mark and his 

mother as they interact at home. 

Types of data Transcripts of interactions. 

Type of analysis Interpretive. 

Results 1. Children learn key as[x-cts of interaction, such as turn- 

taking, gaining the (Icxir, and negotiating meaning very 
early. 

2. The mother does a lot of work to ensure that the 
interactions 'work'. 

3. Pragmatic language functions are acquired before 
prepositional ones. 

4. Children learn to do discourse through interaction, and 
out of this interaction, syntactic structures develop. 



Je: OK, nini. 

R: Ah (protests). 

Je: [Turns out the light] 

R: No! No! No! 

Je: Now it's ni nis time. (No.) Nini. (No.) I'm going to go and get Zoe aiul Jodie 

and Jared and we'll come back, [leaves r(xjm] 

[Silence] 
R: Get. Zoe. Jenny. 

[Silence] 
R: (Sighs) 

[Silence] 

R: I'm a go [.-'] Muuummy! Muuummy! 
Jo: Hello. 

R: A getting out. Jenny a cot a me in. 

Je: See, I got her in the cot. Now, that's ... she can just walk around in the cot for 
her sleep and we'll pretend she's gone to sleep. 

(Author's data) 
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Extnut 4 

( In tl)!*; extract, Kclx'ccn and her mother are talking as Rebecca has her after-sleep 
snack. | 

R: Mm. Wash it. 

M: Washal it - yeah, I washed the tomato. 
R: A hiiinp. 

M: Pardon! niin'l hang It - no. It'll squash. 

Yon hiic it. (Mm) 
R: All 

M: Oh, you've still j;()t a mouthful. Well ciicw it up and swallow it. Chew, chew, 
chew. Chew it on your molars. That's right. How are your molars, good? Oh, 
yeah, hig aren't they. Well now, you going to have a bite of your tomato.' 

R: Alx)lil>lc(lx)ttlo) 

M: Drink. OK. 

(Author's data) 

Kclx-ccn's role i.s very different in these two interactions, as is her language. 
In tlic interactional roiigliliouse in Extract 3, she is on the fjeriphery, margin- 
alized by her status as the youngest interlocutor. She has to work hard to 
IxTonic part of the interaction and have Iicr communicative bids heeded. In 
Kxtratt 4, on the otiicr hand, she is allowed to take the lead. The mother 
supports her and interprets her utterances, as the mother did in the Wells 
transcript, so tliat the interaction proceeds smoothly. The child learns quite 
liilfcrciu things al-H)ut what language can do in both interactions. I would 
argue, not that one context is more appropriate than the other for language 
development, but that the child needs a variety of communicative encounters 
to develop a complete functional repertoire. In terms of research, a complete 
picture of Relucca's linguistic repertoire will emerge only if data for analysis 
are collected in a range of contexts. 

Adult-adult interaction: interpersonal encounters 

III this section we shall look at the work of conversation analysts. The major 
aim of this work has Ix'cii to dcscrilie and ultimately to account for the ways 
ill which coiivers.itions are carried out in socially organised interactions. The 
basic research question aildressed by conversation analysts is: How is it that 
conversational particip.uits are able to produce intelligible utterances, and 
how are they able to interpret the utterances of others (Garfinkel, 1967)? 

As we saw earlier, conversation analysts insist that their analyses be based 
on naturally occurring instances of everyday interaction. They specifically 
reject the use of data obtained through formal experiments, interviews, and 
other forms of elicitatioii, and the use of precoded observational schedules of 
the type we examined in Chapter 5. The rejection of these methods, which 
are widely utiliseil in other areas of social and behavioural science, identify 
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this approach as a separate research discipline. In addition, they reject the use 
of examples which have been invented by the researcher - a practice favoured 
by certain schools of linguistics and applied linguistics. In a key collection of 
papers on conversation analysis, Atkinson and Heritage (1984)point out that 
virtually none of the data in their volume could conceivably be the pr(xiuct 
of recollection or intuition in generating data by comparison with the rich- 
ness and diversity of empirically occurring interaction, 'nor would such 
invented "data" prove persuasive as evidence relevant to the analysis of inter- 
action. Data of this sort can always be viewed as the implausible products of 
selective processes involving recollection, attention or imagination' (p. 3). 
They are just as critical of the use of formal experiments, arguing that the 
success of an experiment will depend on the extent to which the researcher 
has been able to limit, control, and manipulate the behaviour in question. It 
IS the researcher who decides before the event which behaviours arc relevant 
and which are irrelevant. Yet the question remains as to where the variables 
came from in the first place. 

Without previous exposure to a range of naturally occurring interactional data, die 
experimenter is unlikely to anticipate the range, scope, and variety of behavioural 
variation that might be responsive to experimental manipulation, nor will he or she 
be in a position to extrapolate from experimental findings to real situations of 
conduct. By the .same token, while certain of the experimenter's data may or may 
not be artifacts of the more general experimental situation in which the data were 
prixluccd, such influences (if any) can be determined only by systematic comparison 
with a large corpus of naturally occurring data. (Atkinson and Heritage 1984: 3) 

A characteristic feature of this type of research is the elalwrately detailed, 
interpretive analysis of relatively small chunks of language. For example, 
Schegloff (1984) devotes twenty-five pages to a discussion and analysis of the 
use of the utterance 'for whom' in a sixtcen-line interaction taken from a 
nidio 'call-in' show. His focus, in this analysis, is on the ambiguity inherent 
in interaction, and the ways in which such ambiguity is resolved. He dem- 
onstrates that the linguistic form of an utterance does not necessarily coincide 
with the functional intention of that utterance. In the case of 'for whom', 
Schegloff shows that while this looks to be a question, it in fact docs not func- 
tion as a question within the interaction. You might like to consider your 
own interpretation of 'for whom' in the extract, part of which follows. 

B: He says, governments, an' you know he keeps - he talks about governments, 

they sh- the thing that they sh'd do is what's right or wrong. 
A: F\)r whom. 
B: Well he says - he - 
A: [By what standard] 
B: That's what - that's exactly what I mean. 

(Schegloff 1984: 28) 
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A similar type of analysis is carried out by Davidson (1984), who investi- 
gates what happens in a conversation when an invitation, offer, request, or 
proposal is rejected. Her database, which is somewhat more extensive than 
Schegloff s, includes rejection sequences such as the following: 

A: 1 was gonna say if you wanted to you could meet me at UBC and 1 could show 
you some of the other things on the computer, maybe even show you how to 
program, Basic or something. 

B: Well, I don't know if I'd want to get all that involved. 

A: It's really interesting. 

(adapted from Davidson 1984: 108) 

Among other things, Davidson claims to have found that, following a rejec- 
tion, speakers typically reformulate their offer, and that the subsequent ver- 
sion provides the interlocutor with an alternative which provides a face-sav- 
ing way for the interlocutor to reject the offer. 

In these studies, there is no attempt to provide 'representative sainples' of 
the phenomena under investigation. The number of instances of the behav- 
iour can range from a single instance which is analysed in great detail, to a 
dozen or more. A sample from the contents of the Atkinson and Heritage 
collection reveals the sorts of things conversation analysts are interested in 
investigating. 

Orientations 

• On some questions and ambiguities in conversation 
Preference organization 

• Agreeing and disagreeing with assessments: some features of preferred/ 

dispreferred turn shapes 

• Subsequent versions of invitations, offers, requests, and proposals dealing with 

potential or actual rejection 

• Speakers' reportings on invitation sequences 

• Pursuing a response 
Topic organization 

• Generating topic: the use of topic initial elicitors 
The integration of talk iinth nonvocal activities 

• Notes on story structure and the organization of participation 

• Talk and recipiency: sequential organization in speech and body movements 
Aspects of response 

• On the organization of laughter and talk about troubles 

• Public speaking and audience responses: some techniques for inviting applause 

(adapted from Atkinson and Heritage 1984: v-vi) 



Adult-adult interaction: transactional encounters 

In the preceding section, we looked at the analysis of conversations which, 
essentially, have a social function. Similar methods for recording and analys- 
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ing data are used by researchers who are interested in investif^.irmg tr.insac- 
tional interactions. These are interactions whose prim.iry purpose is the 
exchange of goods and services (Merritt 1976). Of particular interest arc 
encounters involving the exchange of professional services. The contexts for 
this research include courts of law, hospitals, doctors' and dciuists' consult ing 
rooms, and so on. Shuy (1973), for example, investigated the interactional 
patterns between medical practitioners and patients. He found that while the 
(Xitential success of medical treatment depended almost entirely on oliraining 
accurate information from patients, physicians in training received little or 
no training in techniques for carrying out diagnostic interviews. 

In a more recent investigation, Candlin, Coleman, and Ikirton (19S3) stud- 
ied interactions between dentists and patients. The focus of their investiga- 
tion was those points in the consultation in which the patients descrilx-d their 
complaint. Building on previous research into doctor-paticiit communica- 
tion within a cross-cultural context, they posed the question: How do 
patients communicate complaints and how do dentists respond? From this, 
they hypothesised that 'dentists and patients bring to consultations ditferenr 
discoursal sets' (p. 59) - that is, awareness by coparticipanrs of what is 
expected/allowable behaviour in an encounter. A patient's perceived lack of 
opportunity to talk is related, not to the amount of actual talk, hut to their 
opportunity to engage in relevant talk, that is, talk related to symptoms. 

The database for the investigation was 66 dentist-patient consultations 
carried out in a range of settings. The data yielded by the investigation are 
illustrated in the following extract. 



Extract 5 

(The patient, female, is already well acquainted with the dentist. .She has Ix-gun 
t.)lking as she enters the consultation RK)in. | 



P (must have something) that was left in and 1 kept thinking it would come 

P out come out because I didn't watit a repetition 
D tight come and sit down 

DSA ( ) 

P I didn't want a repetition of all that bleeding and rushing to the 
D now 

P hospital and restitching and so I just thought well I'll keep persevering 

oh a real old do 

infection and some secondary hemorrhage on it 
(excuse me) 

yes I'm a I'm a real bleeder when the teeth come out always have 
D didn't [ you 

P been and the health hasn't been too good and 1 kept having 
D no that's right 
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D oh you had some er 
DSA 
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I P M ' tlicrc and it er it ( ) 

I n 1 and what's your trouble now there's 

I' j yes 1 think it's an old rwth or something 

I) sonicthinf; brewing tlicrc is [ there 

P that didn't come up didn't come out and I thought wliile there's no trouble 
in the mouth and I've (got) 1 keep getting ulcers in the mouth and I 
thought while it's quiet and cr my chest's better than it's been for a 

P while I'll I ( ) I'll drive in to sec you 

n [ is it yeah riglit let's have a l(x)k now 

I' mm 
I) it's still o-on the same side there 

[6 seconds J 

P [chuckles] 

D that's a piece of a tooth it's not a whole tooth by any 

1 kept lioping it would conic 
a t(K)th 

P fchucklesj | (four years) 

[) I don't know when they had that out [ it's er no it's not 

P ( ) I 

n one I took out for you I've taken nothing as far back as that 

P think it was was it four years this October coming 

1 2 seconds] 

P ah it's alx)ut that time since all I had all that trouble about four years 

P since this Ociolvr 

I) yes ( ) yes but that was nearer the front that was 

P yes yes this has been I've never been to 

D after you'd had teeth out down here 

P anyone since ( ) I've tried lo hope it would work itself out 
I) yes well 

( ) 

one that's been taken 



P 

n mcaTis it's a piece of 



out 

that's conic along ah 



P no 
I) that's not one 1 took out for you it's cr it's 



P is it really | chuckles ( 

n out years ago yes and em 



( ) 

n little piece has come 
through 
(Candlinetal. 198.?: 68-69) 



What is on in tltis intcr.ictioii, and how cati wc tnakc sense of it? The 
discursive analysis provided by the researchers is illustrated in the following 
extract from the research report. The analysis is similar to that used by those 
engaged in aiialysing casual conversation. However, a major difference 
Ivtwccn tiiis work, in which transactional interactions arc analysed, and that 
which looks at the analysis of casual conversation is that the latter carries out 
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detailed analysis on small quantities of data, whereas transactional analysts 
generally work with extensive transcripts. 

Fn this consultation the patient docs not wait to be invited to present her repwrt; 
even before entering the consulting room, she has begun to make her report. The 
patient is typical in that her report is made indirectly, although she is a little 
unusual (but not unique) in reporting on two different aspects of her complaint. 
Thus the patient reports first that "must have something that was left in." That is 
to say, she reports not on the pain or discomfort which she is suffering but instead 
on what she assumes to be the cause of the discomfort, the fact that something was 
"left in." Second, the patient reports on her fears concerning complications 
experienced during previous dental treatment which led her to delay seeking 
treatment again. The precise nature of the complaint for which the patient has been 
delaying seeking treatment is left unspecified. The patient's report is also, clearly, 
highly elaborated. The dentist follows this introduction by presenting the patient 
with an invitation to report on her complaint ("and what's your trouble now") 
which is almost immediately repeated ("there's something brewing there is there"). 
In this way, the dentist ensures that the consultation proceeds according to the 
standard procedure despite its rather unconventional introduction. 

The patient responds to her dentist's invitation by returning to the first of the 
two aspects of her complaint in "I think it's an old tooth or something that didn't 
come up didn't come out." Again, therefore, she reports on the cause of the 
discomfort which she is suffering - the sudden appearance of a nonerupted tooth - 
rather than on that discomfort itself. 

The particular interest of this consultation lies in the dentist's resptjnse to his 
patient's report. The patient has told him that "it's an old ttwth," but in an 
extraordinarily peremptory manner, which the transcript cantiot reveal adequately, 
the dentist declares, "that's a piece of tooth . . . it's not a whole ttxnh by any means 
it's a piece of a tooth." This then is the third technique available for dentists when 
responding to patient's reports: in addition to ignoring reports or minimizing their 
significance, the dentist can openly declare them to be incorrect. (Candlin et al. 
1983: 69-70) 

You can see that this type of analysis consists of an extensive interpretive 
gloss on the interaction. The analysis faces similar threats to reliability and 
validity as other forms of ethnography. Because of the rich database, it is gen- 
erally not feasible to include enough primary data in an article or paper to 
enable an independent researcher to conduct a reanalysis. The studies can also 
face the charge of subjectivity. In the foregoing commentary, for example, 
there are several assertions which wc must take on trust - for example, that 
the beginning of the consultation is unconventional, and that the dentist 
treats the patient in an 'extraordinarily peremptory manner'. Gencralisability 
is also an issue here. The researchers assert that there are three techniques 
available for dentists to respond to a patient's complaint. Elsewhere in their 
study, they claim that the process by which patients report complaints 
involves a three-stage procedure, that dentists give patients only one oppor- 
tunity to present their report, that patients report their complaints indirectly. 



173 



Research methods m language learning 

and that dentists either ignore the report, disagree with it, or agree with it 
hut minimise its significance. As critical readers, we need to ask ourselves how 
many instances of this behaviour would be required in order for us to be con- 
vinced that a general principle of interaction had been identified. We would 
also need to know the extent to which contextual and environmental factors 
influence outcomes. For example, do these principles hold in private clinics 
where the patient pays as well as in public clinics where the state pays? Is 
similar behaviour observed when the patient and dentist are from different 
cultural and/or linguistic backgrounds? Are similar results obtained when the 
interaction occurs in the course of emergency treatment as well as during 
routine dental visits? Finally, we need to ask whether the conclusions drawn 
by the researchers warrant the studies in the first place. 

Thus far, we have focused principally on interactions between native sp)eakers 
(although the studies all have major implications for second and foreign lan- 
guage interactions). In the next section, I shall analyse a major investigation 
of interaction in a cross-cultural context. 

Interaction analysis in a cross-cultural context 

In this section, I shall examine a study that was carried out in order to inves- 
tigate communication tasks in professional workplaces between workers 
from different cultural and linguistic backgrounds (Willing 1990). As with all 
the other studies reported in this chapter, the overall aim of the research was 
to identify the patterns and regularities underlying interpersonal interactions 
in particular contexts. The study was carried out with a group of immigrants 
to Australia, who had professional qualifications and who were undertaking 
an intensive language course designed to prepare them to seek employment 
in their chosen profession. 

The point of departure of the study was the essentially practical one of 
providing guidelines for the development of curricula and materials for help- 
ing immigrants working in professional workplaces. In order to carry out 
these pedagogical tasks, however, it was necessary to obtain descriptions of 
workplace communication tasks, and to identify potential sources of inter- 
cultural communication difficulties. 

The objectives of the study were as follows: 

a) To develop a guiding taxonomy of professional workplace communication tasks 
in terms of encounter type, function, register, channel, and complexity. 

b) To develop conceptual frameworks which can be used to provide language 
teaching practitioners with an understanding of: 

i) a range of communication tasks which commonly recur in professional 
workplace functioning; 

ii) the sub-skills which make them up; 
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TABI.F 8.3 KKSKARCM I'KOCl-.OlIRi; lOK INVKSIKIA riN(; ( :t>MNtHNK:A I KIN IN 
Tin; I'KOI l-SSIONAI, WOKKI'I A( 1 



Step Procedure 



1 Literature review 

2 Initial information collection Ironi selected workplaces 
i Fi)rn)ati<»n ot steering comniitiee 

4 Systematic and extensive data collection from am hemic workplace 
interactions 

5 Data analysis with reference to conceptual moilels developed during 
Steps 1 and 2 

6 Formulation of conclusion and recominencl.ition; compil.iiion ot re|H>rt 



iii) fx)tential source?, ot iiueicultiir^il corniiiuinc.uion breakdown; 

iv) the ways in which various types of encounter Ivtween native speaker and 
non-native speaker (and NNS/NNS) may be made use of for purposes of 
learning. This will involve an examination of how communication irouNe is 
signalled and dealt with hy means of routines such as comprehension and 
confirmaiion checks, ciaritic.nion requesis, and repairs; as well as other 
conscious strategies of coiiimuniciition and learning; 

v) the factors which determine the difficulty and learnahility of specific task 
types encountered in the professioiuil workplace. 

c) To present the above information in ;i research report. 

d) To recommend applications of that information in teaching/learning and 
professional development materials. 

(Willing I WO: 2-.?) 

The decision to investigate native speaker and iion-nativc speaker com- 
munication in professional workplaces was prompted by a change in govern- 
ment immigration policy, which has brought increasing numbers of immi- 
grants with professional qualifications into the country. Many of these 
professionals fail to find employment in the profession for which they are 
trained. This represents a waste of resources for the country and is tieinoral- 
ising for the immigrants themselves. In order to design programs which 
would equip non-native speakers to enter the workforce at an appropriate 
level, it was felt that empirical data were needed on the communication 
demands made on these speakers in their respective workplaces. In essence, 
the overall task of the research team was to characterise the culture of pro- 
fessional workplaces. The procedure followed by the research ream is set out 
in Table 8.3. 

The database for the study came from a groupof students taking an Fnglish 
for Professional Employment course. Part of this course involved spending 
time on work placement. The course participants, with permission from the 
organisations, tape-recorded some of their interactions while on work place- 
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nicnr. Tlicy did this by nctivating a lapel microphone attached to a small cas- 
sette rccortlcr wliich wns cnrricd in tlic participant's pocket or handbag. The 
data yielded hy this procedure were analysed in the same discursive, interpre- 
tive manner as the other studies wc have l(X)kcd at in this chapter. This is 
illustrated in the following sample transcript and commentary. 

Kxtnict 6 

I Tcodoro anil his Australian colleague (who is, in the hierarchy of the company, 

I codoro's tlcpiity ami assistant) arc ciigngcil in a morning meeting in their shared 
office, in wliieli Tcodoro is systematically running through various jobs and other 
matters which have to he ticalt witii. He is, in fact, following an informal agenda 
which he has on paper in front of him (the assistant docs not have a copy of this). 
The following extract constitutes one of the agenda itcms.J 

1 T: Now uli, how do you call the that 

2 special paper that lit in the sliding diK)r.' 

3 (.,: In the sliding diM)r? 

4 1: Sno us 

5 (,': Oh eernphic glass 

6 T: CxTaphiegIa the have anothcrname. Ihat's not 

7 XYZ 

8 (': No this one isn't, Tio. 

9 T: No. Uli these ones to put to install there Ixrfore 
10 the ihirii of Afjril, meaning the XYZ product 

I I (": which Mort Adams is taking 

12 T: Mort Adams is working on it 

13 C: OK 

14 1: Mort, Mort Atlams told me that he is not sure about 

15 it if he have the material or not yet. He will let 

16 me know Kimorrow, which is Wednesday 

17 C: That's right 

18 1: And he is plarwiing to install the new XYZ material 

19 on Thursday. 

20 C:: OK 

21 T: Hut he asked me to clean those one. He told me 

22 that's very simple only to peel off. 

23 C: Yeah? 

24 !': just, just scratch one corner 

25 C: iVel it off 

26 I': AntI then peel off all it 

27 C: OK 

28 T: back, and get it a s|x'cial kind of cleaner. 
I r.xtraet eonliiuies lo line l(X)| 

(Willing 1990: 13) 

A N A I. V -S I .S 

In this segmeiu, the Ix'ginning of the new agenda item is niarketl hy TccKloro's use 
of the marker mwdine I), lie signals the satisfactory completion of the problcm- 
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solving episiKle at line 88 ("thank you, 1 put away") A key factor in this 

interaction is of course the status differeruial between the two participants. 

Teotloro is clearly in charge. This is evident throughout, and is shown from the 
beginning by the fact that he controls the agenda, he provides the transuion, 
topicalises (lines 1-6), and lays the problem situation before his assistant. Tcodoro 
defines the problem situation in lines 9-10 ('these ones to put to install there before 
the third of April'), and describes it in lines 1 1-28. (Willing 1990: 15) 

Having provided data samples, along with an analysis, the task for the 
researchers is to interpret the data - that is, to make sense of it within the 

context of the workplace culture they are seeking to characterise. Of course, 
a large scale study such as this will generate numerous outcomes, and it is 
impossible to detail these here. However, I would like to present one of the 
Hndings to illustrate the way in which the data were interpreted. 

The researchers found that an important construct which helped thein 
make sense of the data was that of the 'meeting'. Most of the interactions 
which occurred, whether they took place in fonnal settings such as board 
rixjins, or informally in corridors, shared characteristics commonly associ- 
ated with meetings. In particular, they were: 

- oriented to a shared, external goal 

- fitted within an explicitly or implicitly agreed upon rime frame 

- had an explicitly or implicitly agreed upon agenda 

- characterised by relatively orderly turn-taking. 

This research has yielded a great deal of valuable data which can he utilised 
in the development of curricula as well as teaching materials and classrcwm 
activities. For example, it has provided us with insights into the generic struc- 
ture and discourse characteristics of 'meetings', which have emerged as key 
interactional events within the culture of the professional workplace. For sec- 
ond language learners, pedagogical procedures based on these insights should 
enable them to enter into, and interact within, the culture of their chosen 
professional workplace. 

Conclusion 

In this chapter, we have seen that the interpretive analysis of naturally cxrcur- 
ring interactions is an important element in research on first and second lan- 
guage acquisition and use. It is particularly significant for those researchers 
who believe that the context in which interactions occur is an important vari- 
able which needs to be taken into consideration. Such researchers arc likely 
to eschew the use of formal experiments, elicitation devices, and other forms 
of intervention in favour of an approach which allows them to investigate 
language use in natural settings. 
As we have seen, in contrast with the experimental method, there arc no 
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highly formalised procedures for collecting and analysing data in this type of 
research. This flexibility can be both an advantage and a disadvantage - an 
advantage in that the researcher can see what is actually there without the 
intervening filter of a methodological procedure, a potential disadvantage in 
that the lack of explicit guidelines may cause the researcher to lose his or her 
way. 

While most of the methodological developments in interaction analysis 
have taken place in investigations of first language use, these techniques and 
procedures are beginning to be used by second language researchers. The 
study reported in the preceding section is an excellent example of theoretical 
models and research procedures deveiofjed in first language research being 
adopted and adapted for use in second language research. 

Questions and tasks 

1. The following extract is a continuation of the interaction between 
Rebecca and her mother. Read the interaction and then complete the ques- 
tions/tasks which follow. 

R: [coughj A Lawrie a bobble, a same a me. 

M: Yes, Lawrie has a cup the same as yours, doesn't he. That's why we have your 

name on the bottom. What's that say.' 
R: P-A-me 

M: P-A-me. Ha! (Yeah) That says B-E-C C-A Becca. 

You're a funny duck. Now, you going to have a bite of your tomato? (Yeah) 
Cause we really have to get going, and we're not taking big tomato in the car. 
No. All over your seat. A big mess. 

R: Got. 

M: Got what? 

R: Num Num. 

M: Piece - piece of num num. Piece of hacon. 
R: Yeah. 

M: Haa - found a piece did you? (Yawns) 

R: Go sleep. 

M: Go to bed. Yes, I do feel a bit sleepy. 
R: (snores). 

M: Why'd you wake up in the night? 
R: A mummy. 

M: Yes, I know that's what you said - 'Mummy'. But why did you want me? 
R: Hub, bub, bub, biy. 

M: Pardon? 

R: Huh, bub, bub, bly. 

M: Huh, bub, bub, hly? (Yeah.) Oh you're making up 
R: Bubble ble... 

M: Bubble a Bill. (Yeah.) You're making up nonsense. No mozzies [mosquitoes], 
were there - when you were in bed last night? No mozzies. 
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R: A duff. 

M: Stuff, yeah I put stuff on you. Cream. 
R: Cream on a mozzie. 
M; Cream on you. 
R: Get me. Oh! Oh! 
M: Mozzies won't get you. 
R: Sore 

M: Mozzies smell that cream and what do they say? 
R: Bzzz!Yuk! 

M: Yuk, they say, and they fly away. 

R- Yeah. Poo, poo, a, a got uuk yuk. Hate bees. 

M: You h.nte bees, do you? (Yeah). 1, 1 don't like bees, cause they stmg. 

R- A ding, (sting) a Jenny, a nioz/.ie a sting a hard a Mummy cuddle. 

M: Yes, I cuddled Jenny, yes. IX) you rc.ncnilx-r when che Ix-e scung you on your 

foot (yes) up at Grandma's (yes). Hurt, didn't it. 
R: Mummy came a ail on a feet. 
M: Mummy came and what, Darling? 
R: A ail a feet. 

M: What did 1 put on your fiKH? 
R* A ail. 

M: Ail? Oil (yeah). Not oil, it was, it was sting cream. Make the sting Ix-ccer. 

R: A tissue. , , , , , 

M: With a tissue, that's right. And go p.it. pat. pat on che Ix-e stmg wuh che cream. 

(Yeah) Yeah. 

R: Oh, mato. ' , 

M: Eat it up, cause we grew that in our garden. T hat s delicious. 

R: Delouse. 

M: Delicious, yes, very tasty. Can you say 'very tasty ! 

R: Very tasty! , 

M: Hee Ha. 1 love tomatoes, specially tomattx-s that we grew in our garden. 

R: Me a a me a ine a sec a Lawrie a dancing. 

M: You and me and Lawrie dancing? 

R: Me, a you a, a, Jenny, Lawrie a dancing. 

M: Dancing? 

R: Yeah. No No (No) No Ahh. 
M: Jumping (No) Dancing? 

M- YouVeft"lck. Well don't have a Ht, because I did.i'r understand you. All righc, 
you try and say it again. Darling. You, and mc and Jenny and Lawrie 

M- DancinJ(No) Dancing (No) Dancing (No). Where do we do it? (No) Oh! (Alili) 
Oh try again. Say it again. Say it again (Ahh!) I>)n't have a ht - you II get 
stuck. You mustn't slide down like that. I'm sorry I didn't understand your 

""^^^^ (Author's data) 



a. Are the comments made by Wells about the interactions lx;tween Mark 
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and liis i!!()thcr pertinent to the interaction between Rebecca and her 
mother? 

b. What do you think Rebecca might have learned as a result of this inter- 
active experience? 

c. Study the transcripts from Rebecca's point of view. What does she already 
know about 'doing' conversation? 

2. To what extent arc the following comments supported or not supported 
by the child-adult transcripts in this chapter? 

When children learn their first language, they do not first learn sounds, then words, 
then sentences, and tlicn finally apply this linguistic knowledge in their interaction 
wirli the social world around them. They begin by interacting with the meaningful 
people in their environment and they converse, play games and engage in rituals 
long lx;fore tliey arc able to utter their first recognizable words. Their caretakers 
typically spend enormous .imoimts of time in setting up and developing these 
interactions, and language develops along the way. (van Lier 1988: 229) 

It can lie olwerved that children learning their first language often plan 
conversations and game-like interaction long before they are skilful users of 
complex syntax. I'arcius and caretakers help them in this planning by providing 
prototypical structures and gradually handing over slots and roles, (van Lier 1988: 
215-216) 

In interactions with children learning their first language, the focus is on the 
mcs.sape lx?ing conveyed, and the vast majority of corrections by caretakers refer to 
violations of meaning rather than form. (Snow and Ferguson 1977) 

I l irsi I langu.i);c acquisition is a sulxonscious process; language acquirers are not 
usually aware of (he fact that they are acquiring language, but are only aware of 
the fact that they arc using the language for communication. (Krashcn 1982: 10) 

3. What possible questions, issues, or hypotheses suggest themselves to you 
as a result of your initial contact with these data? 

4. The following exchange between Rebecca and her mother took place ten 
months after the 'tomatoes' tape. What evidence is there that Rebecca's lan- 
guage has ilevdopcii iti that time? Refer also to Table 8.4. (You can focus on 
morphosyntactic, functional, pragmatic, interactional, etc., features.) 

I R has just had her morning sleep. R and M are putting on one of R's tapes.] 

M: That's the right way. Can you put them away? 
R: Think I need a bit help. 

M: You need a bit of help [ yes] oh right. Where is it? 

No, it doesn't go in that way. No. No. 
R: Where? 

M: Upsi . . . up the other way. That's it. In that way. 
R: Know what one. 
M: Know what one 

I Cassette witli children's .songs and stories begins and continues through segment.] 
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TABLE 8.4 REBECCA; SAMPLE UTTERANCES AT I/ MONTHS AND 17 MONTHS 



27 months 

1. Mm. Wash it. 

2. A bump. 

3. Ah. 

4. A bobble 

5. A Lawrie a bobble, a same a me. 

6. P-A-me 

7. Got. 

8. Num Num. 

9. Yeah. 

10. Go sleep. 

1 1 . (snores). 

12. A mummy. 

13. Hub, bub, bub, bly. 

14. Hub, bub, bub, biy. 

15. Bubble ble... 

16. A duff. 

17. Cream on a mozzie. 

18. Get me. Oh! Oh! 

19. Sore 

20. BzzzIYuk! 

21. Yeah. Poo, fxx), a, a got uuk yuk. 
Hate bees. 

22. A ding, a jenny, a mozzie a sting 
a hard a Mummy cuddle. 



37 months 

1. Think I need a bit help. 

2. Where? 

3. Know what one. 

4. I don't want it outside. 

5. You hear that slurpy noise? 

6. I am. I love ice block. 

7. O ... on a bench. 

8. Yeah. 

9. Baby crying [referring to tape] 

10. Baby crying. 

11. Why crying? 

12. Wants Mummy. 

13. Cause 1 want a drink. 

14. Not really 

15. Cause I want a book. 

16. Cause I want a ... I want a ice block. 

17. Why? 

18. 1 can't rember. 

19. Mmm. Hope you can't do that. 

20. I don't know. 

21. A chicken's man coming? 

22. Mmm, I think I have some spaghetti 
from last night? 



M: Oh, where did you get that from? I said for you to eat that outside. 

R: I don't want it outside. 
M: 1 don't want it inside. 

R: You hear that slurpy noise? 

M: Yeah, I can hear that slurpy noise. Who's doing it? 

R: I am. I love ice bliKk. 

M: Where did you put the bubble gum? 

R: O ... on a bench. 

M: On the bench in the kitchen? 

R: Yeah. 

M: Have you eaten any of it? [No] None? 

R: Baby crying [referring to the tape] 

M: Pardon? 

R: Baby crying. 

M: Yeah. 

R: Why crying? 



181 



Research mcO'^Js in language learning 

M: Why? Oh . . . hungry? Tired? What do you think then? 
R: Wants Mummy. 

M: Wants Mummy, is that why you cry? Why do you cry? 
R: Cause I want a drink. 

M: [laughs] Oh - you don't cry cause you want a drink do you? No. 
R: Not really. 

M: Not really. Why Jo you cry then? 

R: Cause I want a book. 

M: Want a book. ... Not really. 

R: Cause I want a ... I want a ice block. 

M: Cause you ... no, no. not really. Why did you have a fit yesterday? 

R: Why? 

M: I don't know why. I'm asking you. Why did you? 
R: I can't rember. 

M: Can't remember, no. Well, I can remember. You didn't want to come home. 

Remember that? Remember that? 
R: Mmm. (Jumps along with the tape] Hope you can't do that. 
M: I can't do that, no, it's too difficult. My poor old bones. Now, what are you 

going to have for lunch? 
R: I don't know. 
M: Don't know. Sandwich? 
R: A chicken's man coming? 

M: No, chicken man's not coming today. But, you going to tell what you have, 

what you'd like for lunch? 

Mummy'll make you a nice meat sandwich. 
R: Mmm, I think 1 have some spaghetti from last night? 
M: No, we didn't have spaghetti last night. 
R: (inaud) 

M: No, we didn't have spaghetti last night. We had macaroni with the meat. 

(Author's data) 

5. Using the sample data from Rebecca as a point of departure, develop a 
research outline ro investigate some aspect of child first/second language 
development. 

a. Issue/question/hypothesis 

b. Rationale 

c. Subject(s) 

d. Procedure 

e. Type of data 

f. Type of analysis 

g. Measures to guard against threats to reliability and validity 

6. Summarise the Communication Tasks in the Workplace Project by com- 
pleting the follouring: 

a. Question(s)/hypotheses 

b. Rationale 
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c. Procedure 

d. Data 

c. Analysis 

f. Outcomes 

g. Implications 

h. Critique 



Further reading 

Although it has been around for a number of years now. Wells (1981) is an 
excellent and e.xtremely readable intrixluction to child language develop- 
ment, which focuses in particular on the development of intcnicrion.i! skills 
and highlights the importance of interaction to other aspects of child lan- 
guage development. It also contains useful procedural information tor those 
interested in collecting and analysing data on child first or second l.mgu.ige 
development. A standard text on language acquisition is Fletciicr and Car- 
man (1986). Two more recent introductions to child language, which intro- 
duce the reader to a wide range of research, are Reich (1988) .ind l-'oster 
(1990). Finally, Bennett-Kastor (1988) provides a detailed introduction to 
techniques for collecting and analysing children's language. 

For studies of conversation analysis see the edited collection by Atkinson 
and Heritage (1984). The editors also provide an overview of the methodo- 
logical principles and procedures involved in this type of .inaiysis m their 
introduction to the volume. Another relevant collection on conversation 
analysis is Roger and Bull (1989). 

Three excellent introductions to discourse analysis which have k-en writ- 
ten with the classroom teacher in mind are Cook (1989), McCarthy (1991), 
and Hatch (1992). 
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'Either this wallpaper goes or I go'. 

(words attributed to Oscar Wilde on his deathbed) 

Making judgments and evaluations is an integral part of everyday life (even 
toward the end of life if we are to believe the quote above). We are constantly 
evaluating all aspects of our life and work, to such an extent that we rarely 
notice that wc arc doing so. This familiarity can often mask important aspects 
of evaluation when viewed from a formal [x;rs[x;ctive. 

This chapter is .somewhat different in character from the preceding chap- 
ters, nio.st of which have focused on research methodology. In this chapter, 
wc shall look at program evaluation. I have included the chapter for several 
reasons. In the first place, I believe that evaluations, incorporating as they do 
questions, data, and interpretation, are a form of research. Secondly, small- 
scale program evaluations can provide an excellent research training ground 
for graduate students. 

In the first section, I shall define evaluation and attempt to draw a distinc- 
tion Ivtwccn evaluation and assessment. Different types of evaluation arc 
reviewed, and the relation.ship between evaluation and assessment is dis- 
cussed. The chapter also provides a practical guide to the different steps in the 
design of an evaluation. 

Defining evaluation 

In the curriculum litcrnturc, there sometimes appears to be a confusion 
kt ween program evaluation and student assessment. In fact, some writers use 
the terms assessment and evalmtion interchangeably. Gronlund (1981) 
writes: 

F.Viilitatioti may be defined as a systematic process of determining the extent to 
which instructional objectit'es are achieved by pupils. There are two important 
Tispects of this definition. First, note that evaluation implies a systematic process, 
which omits casual, imcontrolled observation of pupils. Second, evaluation assumes 
tli.u imtriu tioiuil objectives have Ixen previously identified. Without previously 
determined objectives, it is difficult to judge clearly the nature and extent of pupil 
learning, (pp. 5-6) 
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Gronlund, in circumscribing evaluation in terms of learning outcomes, pre- 
sents an extremely narrow input-output view of evaluation and, by exten- 
sion, education. In fact, he is using the term evaluation roughly in the sense 
in which I would use assessment. I would like to suggest that, while they are 
obviously related, they mean rather different things - th at evaluation is som e- 
whatbroader in c oncept than assessment. 

To me there is a clear distinction between the two concepts. Assessment 
refers to the processes and proc edures whereby we determine wRat learn ers 
are abTe to do in tfie target language. We may or may not assume that such 
abilities have been brought about by a program of study. E valuation, o n the 
oth er hand, refers to a wider rang e of processes which may or may not include 
assessment data. " ' 



The data resulting from evaluation assist us in deciding whether a course needs to 
be modified or altered in any way so that objectives may be achieved more 
effectively. If certain learners are not achieving the goals and objectives set for a 
course, it is necessary to determine why this is so. We would also wish, as a result of 
evaluating a course, to have some idea about what measures might be taken to 
remedy any shortcomings. Evaluation, then, is not simply a process of obtaining 
information, it is also a decision-making process. (Nunan 1^88: 1 18} 



Two important characteristics of evaluation emerge from this discussion. In 
the first place, it involves not only assembling information but interpreting 
that information - making value judgments. In fact the word value is con- 
tained within evaluation. Secondly, it involves action. We collect information 
about language programs not as a form of philosophical reflection, but in 
order to do something differently next time. Brown (1989) points out that in 
curriculum terms there is a degree of similarity between evaluation and needs 
analysis, which also involves the collection of information for decision- 
making purposes, and that the distinction between the program evaluation 
and needs analysis 'may be more one of focus than of the actual activities 
involved' (Brown 1989: 223). 



The relationship between assessment and evaluation 

While acknowledging a distinction between assessment and evaluation, Hud- 
son (1989) argues that the measurement of student [performance is the key to 
program evaluation. For Hudson, the essential question to be asked by a pro- 
gram evaluator is 'whether an examinee has mastered the content he or she 
has been taught, or has reached a level of com[x;tence defined as mastery' (p. 
259). In his paper, Hudson focuses on what we might call product-oriented 
evaluation. In other words, he is concerned above all else with the outcomes 
of the learning process rather than with the process itself. 
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The researcher who uses assessment data as the key element in an evalua- 
tion has to give careful consideration to three factors. These are ( 1 ) the nature 
of the evidence to be used, (2) the relationship between the evaluation and the 
program goals, and (3) the appropriate measurement instruments to be used. 
The first factor takes us back to an issue which I raised in Chapter 1, that of 
construct validity, and the importance of operationalising the constructs 
underlying one's research. The product-oriented evaluator, no less than other 
researchers, has to be able to define whatever it is that he or she is trying to 
measure - for our purpioses, language proficiency. However, such a definition 
needs to be consistent with the goals and objectives of the program. It would 
be unfair of the evaluator to operationalise language proficiency solely in 
grammatical terms and appply this to the evaluation of a program designed 
to develop interactional skills. 

The other crucial consideration for an evaluator concerned with learning 
outcomes relates to the measuring instruments to be used. Here the evaluator 
must consider whether the instrument is actually measuring what it purp>orts 
to measure. In Chapter 2, we looked at an investigation by Chaudron and 
Richards in which listening comprehension was measured through a written 
cloze test. It has been argued that such an instrument is an inappropriate 
means of measuring general language proficiency (see, for example, Alderson 
1983). In the final analysis, however, the researcher needs to use some sort of 
instrument or assessment procedure for arbitrating between different pro- 
grams, and it is a reasonable bet that someone will criticise one's procedures, 
no matter how carefully they have been constructed. 

The relationship between these three issues - defining constructs, relating 
outcomes to goals, and the appropriateness of the measuring instruments - is 
summed up by Hudson in the following manner: 

The reason much of the discussion centers around the nature of language 
proficiency is that this directly relates to whether specific language tasks or 
program objectives can be specified for instruction and measured validly. This in 
turn relates to the degree to which a test can be used to determine mastery. 
(Hudson 1989:261) 

In selecting assessment procedures and instruments, the evaluator needs to 
be fair to the program being evaluated. This issue of 'fairness' can be a major 
problem when the evaluator is attempting to compare the relative merits of 
two different programs, as Beretta (1986b) shows. In his article, Beretta 
addresses the question of how we can get information for comparative pur- 
poses about the effects of different language teaching programs. The central 
problem here is to find a means of measuring student outcomes which is fair 
to both of the programs being evaluated and does not discriminate against 
one of the programs. He provides the following example of studies which fail 
the test of being program-fair: 
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Asher (1972) and Asher, Kusudo, and de la Torre (1974) invi-stigatal rlic i.'lk-i. t of 
rhe Total Physical Response (Tl'R) tiierh<Kl compared with a "rcfjiil.ir" program. In 
the 1972 report, one of the stories used in classr(X)m training in the TPR group is 
presented as an example; it is entitled "Mr Schmidt gm's to the office." I. at er in tlie 
report, we are informed that one of the criterion mcasiircs hsllI to comp.ire 
experimental (TPR) and control (regular) groups is a listening test involving a 
"story entitled 'Mr. Schmidt goes to the office'" (p. 136). In view of this, ir is hardly 
astonishing that the experimental students drainatically outperformed controls 
(p = .0005). (Beretta 1986b: 432) 

There are strategies which the evaluator can adopt to guard against the threat 
of biasing the evaluation. These include the use of standardised tests, the use 
of specific tests for each program, the adoption of program-specific jjliis [jro- 
gram-neutral ineasures, the identification of common and unique objectives, 
and the appeal to consensus (Beretta 1 986b: 434). The most conunon .strategy 
is to use standardised tests. Because the test items have not Ixrn derived trom 
either of the prograins under investigation, it is assumed that they will not 
provide an unfair advantage to one program at the expense of the other. 
There is evidence, however, that such tests may in fact not Ix" impartial (Val- 
ette 1970). The second strategy is to devise sjjecific tests for e.ich program and 
administer each test to both groups. The problem here is that one can only 
claim superiority for one of the programs if the subjects are .superior not only 
on their own test, but also on the one sp>ecificaHy designed for the other group. 
A compromise between these two fxjsitions would be to devise a test which 
contained program-specific plus program-neutral measures. This third .strat- 
egy was the procedure adopted by Beretta and Davies (1985) in their evalua- 
tion of the Bangalore Communicational Teaching Project in India. The 
fourth strategy suggested by Beretta is to identify the objectives of lx)th pro- 
grams and from these to generate test items that are common to each program 
and unique to each. In criticising this approach, Beretta cites some of the 
objections to the use of behavioural objectives in ciirriciiliini development, 
and points out that the strategy will not be usable if one of the programs is 
not based on behavioural objectives. The final strategy is to appeal to consen- 
sus. Here, the evaluators seek a loose consensus on the construct of language 
proficiency from both parties, and develop testing instruments based on this. 

Beretta 's detailed examination of the options available to the product-ori- 
ented evaluator, and the shortcoinings of each, highlights the dilemma of 
identifying appropriate assessment instruments when one is concerned with 
evaluating the relative merits of two different language teaching programs. 
While admitting to a certain pessimism, he suggests that recent research expe- 
rience has provided us with some basic rules of thumb. 

We know that program evaluation is only a.s good as the criterion measures used; 
that standardized tests are inappropriate toois for comparing programs; and that, at 
the very least, the claims of each specific program must be taken into account in 
test construction if competing interests are to be represented fairly. Whether we 
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complement tliese rough-ami-rcady principles with increasingly elaborate app)cals to 
the nrhitration of "neutrality" or "objectives" or some other tribunal remains to be 
seen. In the meantime, the question of progratn-fair evaluation invites our further 
inspection. (Bcretta 1986b: 441) 

Baciinian (1989) cinims that the role of measurement in program evalua- 
tion has become increasingly unclear in recent years. This is due partly to the 
increasing attention paid to curriculum processes at the expense of products, 
partly to the increasing status of qualitative as opposed to quantitative data, 
and partly to the shortcomings of norm-referenced tests (that is, tests in which 
an iiulividunrs scores arc compared with those of other individuals). For 
Bachman, tlie solution lies in the adoption of criterion-referenced tests, that 
is, tests for assessing what learners are able to do, and in which students are 
assessed, not again.st other students, but according to their mastery of pre- 
specified objectives. 

While information about what learners can do at the end of a program that 
they could not do at the lx;ginning is important for product evaluation, it is 
by no means the end of the .story as far as the evaluator is concerned. In terms 
of internal validity (that is, lx;ing able to state confidently that it was the pro- 
gram which made the difference), one needs a control or comparison group. 
Without such a group, it is impossible to state whether or not the differences 
ohscrvetl in the learners were brougiit about by the prograin or by factors 
external to the program. (It could even be argued that the learners may have 
made even more progress if they had been left alone to pick up the language 
naturally on the street!) 

Unfortimatcly, it is not always possible for the evaluator to create a situa- 
tion in which subjects are randomly assigned to experimental and control 
groups. In fact, it is not always possible for the evaluator to obtain data on 
what the learners could do before taking part in a program. I have been 
involved in several evaluations in which the funding authorities commission 
the evaluation at the conclusion of the program. In such instances, it has beer 
necessary to engage in the comparatively messy and unreliable task of recon- 
structing information .i[x)nt what the learners were able to do before the pro- 
gram Ix-gan from students' notelx)oks, interviews, and other sources. In these 
instances it was also difficult, although not impossible, to obtain control data 
for comparative purposes. This underlines the importance of appointing an 
independent evaluator at the beginning of an innovative program, not at the 
end. While thisexhortation appears consistently in the literature (particularly 
in the reports of post-hoc evaluations), it is just as consistently ignored. 

Program administrators and teachers tend to think in terms of evaluations 
only when their .sources of funding are under threat. In a similar vein, Har- 
grcaves (1989) points out that evaluation is the least well articulated and sup- 
ported dimension of curriculum projects, that it is less often underrated than 
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simply overlooked. Among the many reasons for this state of affairs is 'the 
lack of immediacy and urgency in what evaluation is concerned with; it is 
usually the second volume of the course book or the programme for the next 
in-service course which are needed by tomorrow, rather than the perhaps 
inconclusive assessment of the effectiveness of the first volume of the course 
book or the previous in-service course' (Hargreaves 1989: 35). 

Ideally, product-oriented evaluators need more than pre- and post-pro- 
gram assessment data from control and experiment groups (Long 1984). 
Assessment data will tell us what learners can or cannot do (and, if we are 
lucky, what the learners can or cannot do as a result of taking part in the 
program). However, it will not always tell us why objectives have been 
achieved and why other objectives have not been achieved. In order to make 
such judgments, the evaluator needs access to information about what went 
on inside the classrooms themselves. Long (1980) argues this p>oint in an arti- 
cle in which he refers to the classroom as a 'black box'. Particularly when 
attempting to evaluate and compare two different programs, it is impossible 
to say what made the difference (if indeed there was a difference) without 
access to data about what went on within the classroom. One might also want 
information on a range of other factors and issues which might affect learn- 
ing, such as institutional facilities, the prevailing intellectual and emotional 
climate, relationships between administrative and teaching staff, and so on. 

The importance of process data in program evaluation 

In addition to information on what learners can or cannot do in the target 
language, it is important to obtain data about learning and teaching prcKCSses 
themselves. Systematic observation is one important means of collecting such 
data. Non-observable problems such as failure to activate language out of 
class can be collected through learner diaries and self-reports. Other tech- 
niques, which are described and illustrated in some detail in Nunan (1989), 
include interviews and questionnaires, protocol analysis, transcript analysis, 
stimulated recall, and seating chart observation records. Ideally, a number of 
such techniques and instruments should be utilised in order to obtain multiple 
perspectives on the program under investigation. 

The desirability of obtaining data on program outcomes and teaching pro- 
cesses is illustrated in a study rcfxjrted in Spada (1990). We looked in some 
detail at this study in Chapter 5 and saw that a qualitative analysis showed 
that, while one class spent considerably more time on listening than two other 
classes, the students' listening comprehension did not improve at a commen- 
surate rate. By collecting qualitative classroom data, the researchers were able 
to relate the measurable differences in the listening comprehension of stu- 
dents to the activities and procedures which the teachers themselves 
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employed. If the researchers had only collected pre- and post-intervention test 
scores, the measurable differences would have been uninterpretable. 

T his research demonstrated that there arc in fact measurable differences in 
the way in which instruction is delivered in language programs that have sim- 
ilar ideological underpinnings, and that these differences can be related to 
learning outcomes. On a methodological level, it indicates that we need qual- 
itative data based on classroom observation if we are to interpret, for the eval- 
uative purposes of making decisions about program alternatives, the quanti- 
tative data yielded by assessment instruments of various sorts. If we do not 
have such qualitative data, then it is difficult, if not impossible, to say what it 
was about the instruction and interaction in the classr(X)m which made a dif- 
ference. In the case of no difference ht'mg observed, qualitative data also 
enable the evaluator to say why this should be so. 

Points of focus for program evaluation 

Thus far, I have argued that evaluation is concerned with determining what 
learners have learned from a program, and also with making judgments about 
why instruction has or has not been successful. I have outlined some of the 
problems in obtaining assessment data, and have argued that in order to inter- 
pret such data, one also needs information from a range of other sources 
inside and outside the classroom. While learning outcomes are of paramount 
importance, they are not the exclusive, or even the primary, focus of all eval- 
uations. Having determined that Program A is more effective than Program 
B, one then needs to decide which of the elements within the curriculum is 
responsible for the outcomes. Any elements can form the focus of an evalua- 
tion, particularly limited or focused evaluations, which are concerned with 
part of the total program picture. The types of evaluative questions one might 
pose in relation to different curriculum areas are set out in Table 9.1, which 
has been adapted from Nunan (1988). 

The scope of program evaluation, and the range of issues which can pro- 
vide a fK)int of focus for evaluators, is well illustrated in the studies collected 
in Alderson and Beretta (1992). For example, Mitchell investigates one par- 
ticular approach to bilingual education within an elementary school setting. 
Palmer evaluates a university course for teaching German based on Krashen's 
(1981, 1982) input hypothesis of language acquisition. Coleman focuses on 
the shifting goals within an English as a foreign language project in Indonesia. 
Ross investigates the use of five different methods in the teaching of English 
as a foreign language in Japan. These and the other studies in the collection 
show just how broadly evaluators might be required to cast their net. 

Within the literature on program evaluation, it is traditional to distinguish 
between formative and summative evaluation. Formative evaluation takes 
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TABLK 9.1 St)ME KI;Y QIIHSTIONS IN l'RtH;RAM KVAl llAl lON 



Ctirrii uluiii area 



Sample tiiiestu»is 



The Planning Process 
Needs analysis 



Content 



Intpleinentation 
Methodology 



Resources 

Tc.ichtT 
Learners 



Assessment and 
evaluation 



Are the needs analysis procciliircs effective? 

Do they provide useful information for c-oiirse plaiiniiigi' 

Do they provide useful data on subjective ami ohjeciive 

needs? 

Can the data Ix- translated into coiueiit? 

Are goals and objectives deriveil from neeils analysis? 

If not, from wlu-re are ibcy derived? 

Arc the goals and objectives amjropri.iie lor ilie s|jeeilied 

groups of learners? 
Do the learners think the toiiteiit is appropriate? 
Is the content appropriately graileil? 
D(x.-s the content take speech processing constraints into 

account? 

Are the materials, methods, .iiui activities conson.nit with 

the prespecilied objectives? 
Di> the learners tliuik the materials, methods, and 

activities are appropriate? 

Are rcs»)iirces adequate/appropriate? 

Are the teacher's classroom management skills .ulei|iiate? 

Are the learning strategies of the students ellicient? 
Do learners attend class regularly? 
Do learners pay attention/apply themselves in class? 
I>o learners practise their skills oiitsiile the classroom? 

Do learners appear to be enjoying the course? 
Is the timing of the class and the type of learning 

arrangement sintable for the stiaieiiis? 
Do learners have |>ersoiial problems which interlere vviili 

their learning? 

Are the assessment procedures appropriate to the 

prespecitied (jbjcctives? 
Are there opportunities for self-assesstnent by learners? If 

so, what? 

Are there opportunities for learners to evaluate aspects of 
the course such as ie.irning materials, methodology, 
learning arrangemciH? 

Are there opportunities for self-evaluation by the teacher? 
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plncc during the course of program delivery, and it therefore provides a mech- 
nnism for improving the program during the course of its delivery. Summa- 
tive evaluation, on the other hand, takes place at the end of the program, and 
is therefore not capable of improving that particular program. Summative 
evaluations are carried out to provide information for the modification or 
curtailment of succeeding programs. Both types of evaluation may attempt 
to provide a comprehensive portrait of the program under study or may 
choose to adopt a more limited focus. Akst and Hecht (1980: 264-265) iden- 
tify five key curriculum areas that may form the focus of evaluation. These 
are as follows: 

1. Appropriateness of objective: one may take the position that program objectives 
arc not open to question, since they are presumably the premises on which the 
rest of the program is based. Occasionally, however, an evaluator may take 
exception of objectives that appear to be misguided or unrealistic (for example, 
the objective in a writing program of having foreign students attain the same 
proficiency level as native speakers). 

2. Appropriateness of content to program objectives. 

3. Appropriateness of placement procedure: whatever the bases of the placement 
prcK'cdiire - higii school record, interviews, a battery of tests, or self-selection - 
the procedure itself should be subject to careful scrutiny. Issues meriting 
investigation include content and cutoff scores of a test, and reliability of 
interview ratings, essay scores, and high school grades. Both judgment and 
empirical research play a role in addressing these concerns. 

4. I'ffcctivcncss of instruction: the question here is whether students arc in fact 
learning the content and, if so, whether their learning is the result of instruction 
or extraneous factors. 

5. Klficiency of instruction: can the same learning be provided at a smaller 
investment of time or money? Alternatively, can more learning be obtained for 
the same investment.' 

In addition to formative and summative evaluation, Perkins and Angelis 
{ 1 98.S) dcscrilK norm-referenced, criterion-referenced, and growth-referenced 
evaluation (although they arc using the term evaluation in the sense in which 
I have used assessment, that is, measuring student outcomes). Norm-refer- 
cncctl assessment refers to the practice of comparing one individual's test 
sct)rcs with those of other individuals. In criterion-referenced assessment, an 
individual's performance is compared with some prespecified standard. 
Growth-referenced assessment was developed by those who are critical of 
what is .seen as the arbitrariness of prespecified standards in criterion-refer- 
enced assessment. In growth-referenced assessment, jierformance data are col- 
lected over time, and judgments are made according to whether jjerformance 
goes up or down. In other words, students are judged not against other stu- 
licnts nor against external criteria, but against themselves. Perkins and Ange- 
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lis point out that while there are comparatively few growth-referenced stud- 
ies in ESL, 'Given the disarray and disagreement about standard setting in 
criterion-referenced measurement, it would seem that growth-referenced 
evaluation is a more viable vehicle for ESL/EFL program evaluation, since 
growth is the construct around which all instructional programs are 
anchored' (1985: 86). Whatever the merits of growth-referenced assessment, 
and these remain to be demonstrated, the evaluator is still confronted with 
the task of demonstrating that any growth which occurs is a result of the 
program and not some external factor or factors. 

An evaluation model which was influential in the 1970s and 1980s was 
Stufflcbeam's (1971) CIPP model. Stufflcbeam identified four types of evalu- 
ation - context evaluation, input evaluation, process evaluation, and product 
evaluation. Context evaluation is designed to improve a program by evalu- 
ating and critiquing its strengths and weaknesses. Input evaluation identifies 
the resources appropriate for achieving program goals. With process evalua- 
tion, the focus is on the evaluator providing ongoing feedback during and at 
the conclusion of a program, so that evaluation data may be fed back into, 
and thereby assist in the improvement of, the program. Finally, product eval- 
uation measures the attainments of the programs. These different types of 
evaluation are summarised in Table 9.2. 



Evaluation and research 

A question of central importance to this book is: To what extent can program 
evaluation be considered a form of research? It could be argued that only eval- 
uations designed in such a way as to provide outcomes that are externally 
and internally valid and reliable (such as comparative evaluations of different 
programs) can be considered research. A person holding such a view would 
probably argue that most evaluations making judgments about a single pro- 
gram are not research because they lack external validity. They do not pur- 
f)ort to provide data beyond the research site where the data were collected. 

I take a broader view, and have argued throughout this book that any 
investigation which contains questions, data, and interpretations of the data 
qualifies as research. I would therefore accept that evaluations, even those of 
a single program, are, in fact, research. While evaluators who are investigat- 
ing a single program can usually ignore issues relating to external validity, 
they still have a responsibility to guard against threats to the internal and 
external reliability, and also the internal validity of their investigations. Inter- 
nal validity can be particularly problematic for research conducted in a field 
setting. According to Airasian, there are eight major threats to the internal 
validity of field research. These are set out in Table 9.3. (See also the discussion 
on validity in Chapter 3.) 
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TABI H 9.3 Tllid- AIS TO 7KF. INTERNAL VALIDITY OF FIELD RKSKARCH 



Threat 



C.oiiiDwnt 



l listory Hvents external to the program, policy, or practice under 

investigation wliicli occur Ix'twccTi pre- and posttesting 

Maturation Natural biological, psychological, or sociological development 

of subjects occurring between pre- and fwsttcsting 

Instability Unreliability of measures, which causes fluctuation in scores 

independent of the program, policy, or practice under 
investigation 

Testing Mie effect of tnking a tost at one point in time upon taking 

the test at a subsequent point in time 

Instrimientation The effect of changes in the measuring instrument between 

pre- and posttcsts so that observed effects are a result of the 
instrument change and not the program, policy, or practice 

Selection Biases resulting from differences between types of individuals 

recruited for comparison groups 

Mortality Differential loss of subjects from comparison groups 

Statistical The effect of selecting individuals on the basis of their 

regression extremely high or extremely low scores on a measuring 

instrument. Scores at the extreme ends of a distribution are 
unreliable, and retesting tends to result in either extremely 
high scorers scoring relatively lower, and hence 'regression' 
downward in test scores, or extremely low scorers scoring 
relatively higher, and hence 'regressing' upward in test scores. 
Unreliability of test scores at extreme ends of the distribution, 
in and of itself, results in changes in scores upon retesting. 



Source: After Airasiaii (1974: 162-163). 



Elements in the design of an evaluation study 

In this .section, I shall look at some of the practical issues and problems which 
need to be dealt with when designing an evaluation. The discussion will be 
or};aniscd around the following key questions: 

- What is the purpose of the evaluation? 

- Who is the audience for the evaluation? 

- What principles of procedure should guide the evaluation? 

- What tools, techniques, and instruments are appropriate? 

- Who should carry out the evaluation? 

- Wiicn siiould it be carried out? 

- What is the time frame and budget for the evaluation? 

- How should the evaluation be reported? 
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PURPOSE 

As with other types of research, it is extremely important to clarify from the 
beginning the aims and objectives of the evaluation. This is not always easy 
to achieve and can involve considerable negotiation, particularly when there 
are numerous interest groups involved. Teachers, administrators, parents, 
funding authorities, and learners will often have quite different perceptions 
of the purpose of an evaluation. If these are not dealt with satisfactorily and 
put in writing before the evaluation begins, then other steps in the evaluation 
process, such as designing data collection instruments, will be almost impxjs- 
sible to carry out. There is also the danger that, at the end of the evaluation, 
the outcomes themselves will be unacceptable to one or more of the interest 
groups. 

If the purpose of the evaluation is to provide information for the ongoing 
improvement of the program, it will be formative in nature. On the other 
hand, if the purpose is to provide information for accountability, then it is 
more likely to be summative. Whether it is to be formative, summative, 
or both, the evaluation needs to be planned for from the beginning of the 
project. 

AUDIENCE 

Audience is an important preliminary consideration, and one which is not 
always given adequate attention. As indicated earlier, different audiences will 
often perceive different purposes. Funding authorities will want to know 
whether their money is being spent wisely, and may look to a summative 
evaluation which measures what learners have or have not achieved as 
a result of taking part in the program. Teachers, on the other hand, 
may want a rich, interpretive account of the program which is formative 
in nature, non-judgmental, and reflective of multiple perspectives and 
interpretations. 

An additional danger, and one which Hudson points out, is that the audi- 
ence may change during the course of the evaluation, or even at its conclu- 
sion: '. . . evaluations will a|ways have audiences, and the original audience 
seldom remains the sole audience. Peer evaluations made for a teacher trainer 
have a way of coming to the attention of the program director. An evaluation 
for the program director has an uncanny way of being quoted later by some 
funding agency' (Hudson 19^9: 259). The fact that an evaluation re[X)rt may 
fall into the hands of those ibr whom it was not initially intended places an 
additional burden on the researcher. 

Hargreaves (1989) also suggests that the evaluator needs to consider 
whether the target audience is a specialist one, versed in the subject domain, 
or whether it consists of generalists who will not necessarily be familiar with 
the key concepts associated with the field or area being evaluated. 
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I'RINCII'LES OK PROCEDURE 

A major practical impediment to the successful completion of an evaluation 
may be a disagreement on the part of those closely involved in the evaluation 
on the scope, nature, purpose, and respective rights of those involved. Each 
of the key players in an evaluation, be they teachers, learners, administrators, 
bureaucrats, or the evaluators themselves, will bring their own perspectives, 
sets of beliefs, and realities to the evaluation. Lfniess clearly articulated prin- 
ciples of procedure are laid out at the beginning of the evaluation, it is quite 
{xjssible that these differences of perception may jeopardise the evaluation. 
One way of preempting this is for the parties involved to negotiate a set of 
principles of procedure which can be agreed to at the outset of the project, 
and subsequently drawn upon in cases of disagreement. The following com- 
prehensive set of principles were drawn up to guide the evaluation of a major 
curricular innovation. They provide a clear statement on the relative rights 
and responsibilities of the participants in relation to evaluation data as well 
as outcomes and recommendations. 

1. No participant in the project will have privileged access to the data of the 
evaluation. 

2. No participant will have a unilateral right to, or power of veto over, the 
content of the report. 

,3. The evaluators will attempt to represent the range of viewfjoints encountered 
in the evaluation. 

4. Explicit and implicit recommendations made by the evaluators will not he 
regarded as prescriptive. As far as possible, recominendarions will reflect the 
views of the participants, nor the evaluators. 

5. The evaluators will assume that they can approach any individuals involved in 
the project to collect data. 1 hose approached should feel free to iliscuss any 
matter they see fit, and all such discussions will be treated as confidential by the 
evaluator. 

6. The release of specific information likely to identify informants will be subject 
to negotiations with these informants. 

7. The criteria of fairness, relevance and accuracy form the basis for negotiation 
between the evaluator and participants in rlic study. Where accounts of the 
work of participants can he shown to he unfair, irrelevant or iii.icciir.itc, the 
report will be amended. Once draft reports have been negotiated with the 
participants on the basis of these criteria, they will be regarded as having the 
endorsement of those involved in the negotiations with regard to fairness, 
relevance and accuracy. 

8. There will be no secret reporting. Reports will be made available first to those 
whose work they represent. 

9. Interviews and meetings will not be considered 'off the record', but those 
involved will be free, both before and after, to restrict aspects of parts of such 
exchanges, or to correct or improve their statements. 

10. The evaluators are responsible for the confidentiality of the data collected by 
them. 
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TAHl K 9.4 I VAI IIATOR S KOl.i:, ( A I l-CJOK I l-.S, AND IMUH l DDIil S 



Ei'dltiJtor's role 



Outsider looking in 



Facilitator drawing out 
information 



C-Jtciiories 



Kxisting infornuition 



Tests 



Oliservations 



Interviews 



Meeting 



Questionnaires 



I'rikwliirci 



- Records analysis 

- Systems analysis 

- Literature review" 

- Letter writing 

- I'rt)ticicncy 

- riaccmeni 

- Di.ignosiic 

- Achicvcnicnl 

• Clase .sliiilies 

■ Diary stmlies 

' Behavioral olwerv.nion 

- Interactional analyses 

■ Inventories 

' Individual 
Group 

IX-lplii tcchniiiue'' 
Advisory 
Interest group 
Review 

Kiodata .surveys 
Opinion .surveys 
Self-ratings 
Judgmental ratings 
Q sort 



■"A written summary and critique of research relating to a question or issue. 

'a procedure in which a set of questions or tasks is sent out 10 experts in the (iclii. 

The collective responses are then collated and analysed. 

Source: Reprinted with permission from Brown (1989: 2.3.3). 



(L. Bartlett, 1988. 'Proposal for the evaluation of the National Curriculum 
Project', Adult Migrant l-ducaiion Program, Australia, iinpublislieil 
manuscript.) 



TECHNIQUES AND INSTRUMENTS 

Most of the discussion in earlier sections of this chapter has concerned the 
place of assessment data in evaluation, although it was made clear that assess- 
ment data alone are inadequate for the purposes of program evaluation. In 
fact, any of the data collection methods dealt with in this book can Ik* used 
by the evaluator, as can be seen in Table 5-4, which is taken from a practical 
synthesis of existing possibilities in program evaluation by Brown (1989). It 
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can be seen from the table that the cvaluator can either be an outsider or some- 
one who plays tiie role of a facilitator within the program being evaluated. 
The table also illustrates the wide range of instruments and techniques which 
the evaluator can draw on for collecting data. 

DATA ANALYSIS 

A basic question flowing from the choice of techniques and instruments con- 
cerns the types of analyses to be used and whether these are to be statistical, 
interpretive, or both. Evaluations have the potential to yield huge quantities 
of data, and decisions need to be made on how the qualitative data resulting 
from semi-structured interviews and questionnaires are to be reduced to man- 
ageable proportions. 

TIMI-: l-RAMU AND BUDGET 

F.stahlisliing a realistic time frame and budget are important for the successful 
completion of the project. In those cases where one is competing for outside 
funding, the success of one's bid will often rest on the time line and budget, 
and whether these fit in with the time line and budget of the funding body. 

In carrying out an evaluation for an external body or agency, there arc 
generally nine essential steps: 

1 . Become familiar with the project, program, or innovation. 

2. Submit an expression of interest. 
.1 Prepare an evaluation proposal. 

n. identify the objectives 

b. determine the data collection methods 

c. determine the data analysis methods 

d. identify the subjects 
c. ft)rmulate a budget 
f. establish a time line 

4. Kefine the evaluation proposal - consultation and negotiation with the 
clients. 

5. Develop, pilot, and refine the data collection instruments. 

6. Identify and negotiate the data collection sites. 

7. Analyse the data. 

8. Prepare and circulate a draft report. 

9. Kevise the report and submit a final version. 

RF.PORTING 

The reporting process can take much longer than anticipated, particularly if 
there is disagreement by funding authorities on points of fact, outcomes, or 
recommendations. It is important for those who commissioned the evaluation 
to he given a draft of the evaluation report with the opportunity to comment 
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TABLE 9.5 CENTRAL QUESTIONS IN PROGRAM EVALUATION DESIGN 



Element 



Questions 



4 



Purpose 
Audience 



Principles of 
procedure 

Techniques and 
instruments 

Data analysis 

Time frame 
and budget 



Reporting 



What is the purpose of the evaluation? 
Is it basically formative or summative in nature? 

What is the audience for the evaluation? 

Are the interests of the different stakeholders compatible 

or incompatible? 

Docs the audience consist of specialists or gencralists? 

What principles of procedure should guide the evaluation? 
How should these be negotiated between those involved in 
and affected by the evaluation? 

What techniques and instruments are to be used? 

Are these appropriate to the aims of the evaluation? 

Do they imply an insider or outsider role for the evaluator? 

Will the data analysis be statistical or interpretive? 
Is the data analysis within the scope of the evaluation? 

Is the budget adequate given the evaluation brief? 
Does the time frame incorporate all the essential steps 
needed for the evaluation? 
Is the time frame consistent with the budget? 

How is the evaluation to be refKjrtcd? / 
Does the evaluation plan allow for drafts of the rep>ort to 
be negotiated with those funding the evaluation? 



on it before it is released. If the funding authorities are not happy with the 
report, then there is a danger that the final refxjrt may not see the light of day. 
This negotiation process can greatly extend the evaluation time line. 
The points and issues raised in this section are summarised in Table 9.5. 



Doing evaluation: a case study 

In this section, I shall provide an illustrative summary of an evaluation which 
was carried out recently within a school district in Australia. The general pur- 
fxjse of the evaluation was to provide information to the educational author- 
ities who had funded an innovative in-service package designed to improve 
the writing skills of students in a cluster of 'disadvantaged' inner city ele- 
mentary schools. 

TITLE 

An evaluation of the Disadvantaged Schools Project Writing Package. 
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OBJECTIVES OF THE INNOVATION 

The objectives of the innovative package were as follows: 

1. That students demonstrate improvements in their ability to respond effec- 
tively to the writing demands of the curriculuni. Specifically, they will be 
able to perform effectively in written class assignments, make effective 
notes, do independent research, complete written homework assignments, 
and participate fully in classroom discussions about writing. 

2. That students understand the criteria by which their writing is being 
assessed and act to meet these criteria in their writing. 

3. That the positive impact of the teaching/learning cycle, known as the 
'curriculum genre', on students' verbal and reading abilities is 
demonstrated. 

4. That teachers participating in the genre writing package be able to iden- 
tify examples of the following genres: recount, report, procedure, expla- 
nation, exposition, discussion, and narrative. 

5. That teachers will be aware of the significant language features of the 
genres listed in Objective 4. 

6. That teachers will be able to apply their knowledge of genre theory to 
identify the schematic structures and significant language features of 
genres other than those identified in Objective 4. 

PURPOSES OF THE EVALUATION 

There were three principal purposes of the evaluation relating to the impact 
of the innovation on both teachers and learners, and the elements within the 
package which had and had not proved to be effective. The purposes were 
summarised as follows: 

1. To assess the impact of the writing package on students' writing. 

2. To evaluate the impact of the package on teachers' 

a. capability to assess the effectiveness of students' writing 

b. pedagogy 

c. knowledge of the social functions of language. 

3. To identify which elements of the package have been most beneficial and 
which require amendment. 

DATA COLLECTION METHODS 

Four principal data collection methods were used in the evaluation. These 
were: 

1. A detailed questionnaire completed by teachers involved in the project 

2. Focussed interviews with teachers and other key jjersonnel 

3. Observation and analysis of lessons 
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4. Analysisof students' writing samples from schools taking part in the inno- 
vation and also from schools not involved in the innovation. 

TIME LINE 

In the initial brief it was envisaged that the project would take seven months 
to complete. The project plan and budget were baseil on this rime line, .i sum- 
mary of which follows. (Note that the project ttx)k several months more than 
this.) 

May Appoint principal researcher 

Devise questionnaire and distribute to schools 
June Interview consultants and authors of in-service pack.ige on 

goals, nature, and implementation of package 
Conduct literature review (see glossary) 
Identify non-package schools to act as control 
Collect samples of students' writing from package and non- 
package sch(X)ls 

July Collate responses to questionnaire and select school for furtlier 

evaluation 

Conduct structured interviews with teachers, students, and 
parents 

Record sample lessons and collect written texts relating to these 
lessons 

August Complete interviews, recordings of sample lessons, and collec- 
tion of texts 

Begin data analysis and evaluation of students' writing 
September Complete data analysis and evaluation of .students' writing 

Begin drafting report 
October Submit draft report 
November Revise report 

Submit final draft 

TYPE OF DATA AND ANALYSIS 

The evaluation resulted in three types of data - interview protocols, ques- 
tionnaires, and samples of students' writing. The major analytical tasks con- 
fronting the evaluators were the comparative analysis of the 1,5(X) pieces of 
writing, and the collation and analysis of the questionnaire data. 

ANALYSING STUDENTS' WRITING 

Assessing the quality of written language is a complex matter, and the criteria 
by which one carries out such a task will depend on one's beliefs about the 
nature and purpose of language. For example, if one believes that the creative 
use of language is important, then one will evaluate positively writing which 
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TABLE 9.6 SCHEMATIC STRUCTURE OF SUCCESSFUL TEXT 



Structure 


Clauses 


Text: 'The Skull and the Skeleton ' 




1 


One day there was a poor orphan girl 


Orientation 


2 


She had to work with her stepmother 




3 


u _ 1 J • ^ 1 • 1 I 

Her hands were going to skin and bones 




4 


So she decided to run away 




5 


She saw a castle 




6 


II II ^ 

So she knocked on the door tap tap tap 




1 


A skull with no body opened the door 




Q 
A 


and he said yes 


Complication 




1 nc Cin lOiu inc skuii |_wndi nau nappciicu lu iici j 


in 






1 1 


Shf* ^ \\cu\v without a ^kull 




12 


She knew thnt it belonged to the skull 




13 


And the skull told the girl [what had happened]' 




14 


The they [sic] had dinner 




15 


She stayed two night [sic] 


Rcsoliiiion 


16 


;llid she kissed the skull 




17 


They got married 




18 


They lived happily after 



'The words in brackets were actually written by the student. 



exhibits characteristics such as figurative and metaphorical images, the use of 
fantasy, and so on. in the present instance, the evaluators selected criteria 
which were consistent with the linguistic model underlying the writing pack- 
age. In this model, the essence of successful writing lies essentially in the over- 
all structure of a text, its development and cohesion, and whether or not it is 
written in language appropriate for its intended purpose and audience. The 
criteria not only had to reflect this view of 'good writing', but it had to be 
practically applicable within the limitations of time imposed by this evalua- 
tion (Walshcctal. 1990: 18). 

Following the model, three criteria were selected for analysing the data: 

1. Is the schematic structure appropriate for the genre of the text? (In the 
model, it is argued that texts written for different purposes will exhibit 
different patterns of overall organisation and text structure.) 

2. Docs the writer explicitly identify the topic, and was the topic developed 
appropriately.' (If the writer fails to develop the text topic or switches from 
one topic to another, then the text is confusing and difficult to follow.) 

3. Does the writer use reference appropriately? (Appropriate use of reference 
is an indicator of text cohesion and an indicator that the writer has 
a sense of the 'decontextualised' nature of writing in comparison to 
speaking.) 
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TABLE 9.7 SCHEMATIC STRUCTURE OF UNSUCCESSFUL TEXT 



Structure 


Clauses 


Text: 'Aboriginal Skeletons and Skulls' 


Thesis 


1 


All around the world the museums do need some 


(statement) 




skeletons and skulls 


Argument 1 


2 


Well the Aboriginals gave them some of their 


(statement) 




grandparents to put in the museum's 


Argument 2 


3 


Well Loir Richards is an Aboriginal 


(recount) 


4 


and she said that some people say 






that Aboriginals have not got any feelings 


Conclusion 


5 


The skeletons and skulls should go back 




6 


where they come from and remain 


Argument 3 


7 


You would not like it 




8 


if they took your grandparents skeletons and skulls 



The sample texts set out in Tables 9.6 and 9.7 illustrate the way in which these 
criteria were applied to an analysis of the texts. The successful text is a nar- 
rative, the schematic structure of which is as follows: ^ 

- orientation (the context for the story is provided and the major partici- 
pants are introduced) 

- complication (the problem or point of the story is presented) 

- resolution (the problem is resolved) 

- evaluation (the writer comments on the narrative) 

Commentary on text: 

[This text] like many other narratives collected for this analysis contains no 
evaluation and reveals that this young writer, like many others, lacks full control of 
the narrative genre. However, for the purfXKe of this evaluation of young children's 
writing the essential stages of the narrative have been taken to be orientation, 
complication and resolution, and hence is assessed as satisfying criterion 1. The 
topic of [the text] is developed in the sense that the adventures of the 'poor orphan 
girl' are related to the meeting of the 'skull' and subsequent finding of its 
disengaged body. While the logical sequence of some events in the narrative such as 
the skull telling 'what had happx;ned' and 'having dinner' are not especially clear, 
there is enough information about the skull and skeleton for the reader to follow 
both the sequencing of events and the connection between complication and 
resolution. Hence the text is considered successful in terms of criterion 2. Reference 
is used appropriately in the text. The major participants are explicitly introduced: 
'a poor orphan girl', 'a skull with no body' and thereafter referred to appropriately; 
'she', 'the girl', 'he', 'the skull'. Thus it is clear at all times who or what is being 
referred to in the text. (Walshe et al. 1990: 20) 

The unsuccessful text was produced after the writer had viewed an educa- 
tional program on Aboriginal remains. The writer's task was to argue the 
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question of whether or not these remains should be preserved in museums or 
whether they ought to be returned to Aboriginal people for burial. The text 
is an exposition and has the following schematic structure: 

- thesis (the argument is introduced) 

- arguments (the thesis is supported) 

- reiteration of thesis 

Commentary on text: 

. . . the structure u( Text 2 can be summarised as follows: a general statement, 
which could be generously interpreted as a thesis, followed by a second statement, 
rather than an argument. Next is a short recount rather than a second 
argument. . . . [This isj followed by the writer's conclusion regarding this topic and 
then there is a follow up argument, h is the conclusion that gives the clearest 
indication that the writer intended the text to be an Exposition. Other stages in the 
text are not those of a successful Exposition. . . . Development of topic in Text 2 is 
unsatisfactory. While the text is loosely cohesive around the topic of museums and 
skulls and skeletons, there are problems in that none of the arguments follow 

logically from one to another There are alsti minor problems with referencing. 

The opening statement refers to 'the museums'. Such reference is unclear as the 
reader is not informed which museums are being referred to. . . . Thus Text 2 is 
assessed as unsatisfactory on all three criteria. (Walshe et al. 1990: 21) 

ANALYSING QUESTIONNAIRE DATA 

The questionnaire contained thirty questions, twelve of which were closed, 
and eighteen of which were open ended. The open-ended questions yielded a 
great deal of qualitative data, and the researchers were confronted with the 
problem of quantifying these in a way which enabled patterns to emerge. 
They did this by constructing a series of analytical categories from the state- 
ments themselves. For example, the questionnaire probe which instructed 
teachers to 'stare three beliefs you have about language development that 
determine the way you teach' resulted in several hundred statements. On 
analysis, it appeared that these related to one of three broad concepts - lan- 
guage/learning, the environment/climate, or the learner. These major cate- 
gories were further subdivided, and the statements were assigned to the 
appropriate category. (The results of this analysis can be found in Chapter 7, 
where we looked at ways of quantifying qualitative data.) 

One of the problems confronting the evaluators concerned the interpreta- 
tion by teachers of the model underlying the package. They attempted to gain 
insights into teachers' belief systems indirectly, by posing questions such as 
the following: 

Question 14 

Assume the following scenario. 

Your school participated in the Writing Package in 1990. It will enter a second 
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TABLE 9.8 teachers' aovic:k TO t:oi.i.i;Ac;iii;s ON I'Ac KACii;: analv 1 u al 

CATEC;ORIlS 


Category of teat fier ailince 


N 


Reference to genre/nnxlel/factual writing 


61 


Reference to handb<M)k/().lckage/vide<>/siipij()rt rn.neri.il 


A\ 


Reference to consultants 


21 


No comment 


19 


Reference to practical nature of p.R'k.ige/delinite procedures 


IS 


TcH) early to conHiieiu 


16 


Reference to pt)sitive reaction of chililreii 


15 


Reference to purposeful nature of writing 


12 


Reference to complexity /difticulty of nuKiel/package/iieed to |)ersevere 


11 


Reference to shortcomings of process writing/ previous approaches 


9 


Reference to spoken/ written language rel.itionship/contrast 


6 


Reference to specihc practices, e.g., )omt construction 


7 


Empowering nature of package 


7 


Language across the curriculum 


7 


Reference to beneficial effect on writing 


.1 


Total 


256 



stage t)f participation in 1991. A new teacher previously not involved will siari at 
your schtxil in 1991. Write some brief notes to him/her in which you descrilx- the 
content of the package and in which you descrilv your reaction to the conteiit. 

Once again, this strategy resulted in a great deal of qualitative data whicli 
had to be quantified in some way. In this particular case, the tliscursive notes 
made by the teachers were scanned for key terms which were used to create 
analytical categories. Statements were rhcn assigned to the relevant catego- 
ries. The result of this atialysis is set out m Table 9.8. 

OUTCOM ES 

The evaluators came to the following conclusions: 

1. The innovation had an overall positive response from participating teach- 
ers, with teachers giving an overwhelmingly positive response to the writ- 
ing package. 

2. The innovation had a beneficial impact on students' writing. A conipari- 
tive analysis of texts from package and non-package schools indicated 
that, in terms of the evaluation criteria identified by the researchers, stu- 
dents in package schools produced a greater range of factual texts, aiKi 
produced them more successfully. 

3. Teachers' classroom practices changed as a result of participation in the 
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jjiiiovntioM. Although teachers incorpordted ideas from the package into 
their teaching, they adapted these significantly to suit their needs. 

4. The objectives of the innovation were largely met. 

5. The model of in-service upon which the innovation is based has a number 
of distinctive features which contributed to its effectiveness. These include 
the balance of theory and practice, the demonstration lessons, and the 
cyclical nature of the input. 

C:R ITIQIJI- 

The riuijor structural shortcoming of the study stemmed from the fact that 
the evaluation was not commissioned until the innovation had concluded. 
This m.ade it impossible to collect pre- and post-innovation writing samples 
and teacher comments from package and non-package schools. Despite the 
fact that the literature on evaluation is replete with exhortations to program 
innovators to build an evaluation component into their projects, this seems 
to be relatively infrccjucnt. The lack of pre-intervention writing texts calls 
into question the internal validity of this study. 

Despite the care with which the analytical categories were constructed for 
the analysis of the free-form comments from teachers, there is always the pos- 
sibility that the categories and constructs arc artifacts, and either misrepre- 
sent or distort in some way the original data. The researchers attempted to 
minimise this threat by the involvement in the analysis of all four members 
of the evaluation tean7. 

There is a question as to the external validity of the study, since we do not 
know the extent to which the results can be applied beyond the groups inves- 
tigated. The evaluation was focused on elementary students in a number of 
urban schools which had Ix'cn dcsignatcil as 'disadvantaged' by government 
fiat. I o what extent the innovation would be appropriate for pupils in schools 
not designated as 'disadvantaged' or to other levels in the system, such as sec- 
ontlary students, must remain doubtful. 

Ncitwitlistanding the shortcomings in the design and execution of this 
evaluation, it represents a carefully conducted piece of research. The research- 
ers piloted their procedures and instruments, including the questionnaire, 
and ihcy collected ilata from multiple sources. Most important of all, they 
were guided throughout the evaluation by the theoretical model on which 
the innovation was based. 

Conclusion 

Program evaluation is an enormous subject, and many volumes have been 
written alx)iit the c()nce[)tual and practical aspects of doing evaluation. In this 
chapter, I have trieil lo give some indication of the scope of program cvalu- 
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ation from the perspective of research. 1 have justified the inclusion of this 
chapter in a book such as this on the grounds that program evaluations count 
as research, since they contain questions, data, and interpretive analysis. 
While such a claim may be said by some to stretch the definition of research 
to the breaking fx)int, I believe that it is justified. 

The chapter reviews recent writing on evaluation in language education. I 
have dealt with the scope of evaluation from the perspective of the language 
curriculum developer, and highlighted some of the practical problems asso- 
ciated with the collection, collation, and analysis of evaluation data. In the 
process, I have shown how any element within the curriculum may be eval- 
uated. Processes of program evaluation are illustrated with an extended 
example from a recent study into an innovative program in a second language 
context. 

Program evaluations are particularly suitable as training vehicles for grad- 
uate students. They present a ready-made area for investigation, the scope of 
the evaluation can be tailored to the time and resources available to the stu- 
dent, and there are numerous models and procedures available in the pub- 
lished literature for collecting and analysing data. 

Questions and tasks 

1. An increasingly pwpular means of assessing achievement is the proficiency 
rating scale. The following is the generic description of foreign language 
sfjeaking proficiency at high intermediate level. It is taken from the American 
Council on the Teaching of Foreign Languages Provisional Proficiency 
Guidelines. 

Able tosalisfy most survival needs and limited social demands. 

Shows some spontaneity in language production but fluency is very uneven. 

Can initiate and sustain a general conversation but has little understanding of the 

social conventions of conversation. 
Developing flexibility in a range of circumstances beyond Immediate survival needs. 
Limited vocabulary range necessitates much hesitation and circumlocution. 
The commoner tense forms occur but are infrequent in formation and selection. 
Can use most question forms. 

While some word order is established, errors still occur in more complex patterns. 
Cannot sustain coherent structures in longer utterances or unfamiliar situations. 
Ability to describe and give precise information is limited. 
Aware of basic cohesive features such as pronouns and verb inflections, but many 

are unreliable, especially if less immediate in reference. 
Extended discourse is largely a series of short, discrete utterances. 
Articulation is comprehensible to native speakers used to dealing with foreigners, 

and can combine most phonemes with reasonable comprchcnsibility, but still has 

difficulty in producing certain sounds in certain positions, or in certain 

combinations, and speech will usually Ix" labored. 



209 



Research methods lu Unguage learning 

Still hjs to repeat utterances frequently to be understotxl by the general public. 
Able 10 produce some narration in either past or future. 

(cited in Freed 1984: 228-229) 

a. How migbt such a scale be used as part of an evaluation? 

b. What do you see as the probleriis of using such a scale? 

c. What additional/alternative data would you collect? 

2. Prepare an evaluation brief for a curriculum innovation with which you 
are familiar.. Include the following information: 

- Brief description of innovation 

- Evaluation questions 

- Type of data 

- Type of analysis 

- Time line and budget 

- Anticipated problems 

- Possible solutions to problems 

Further reading 

As I have already indicated, there is a great deal of material available within 
the educational literature on program evaluation. For the language teacher, 
the best place to begin is with the papers on evaluation published in Johnson s 
(1989) collection on second language curriculum development, and then to 
read the studies in the collection edited by Alder&on and Beretta (1992). Prac- 
tical guidelines for the evaluation of language education are provided in Ald- 
erson (1992). The papers by Beretta (1986a, b) are also important contribu- 
tions to the field. For an excellent case study of an educational innovation, 
see the evaluation of the Bangalore Project by Beretta and Davies (1985). 
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Wise folk niny or ni.iy not form c.'<(>ect;nii>ns nboiit wii.it the tiiliire hulJs in siure 
but the fiK)lish can Ix.- relied u\Kin to predict with toniplete conliilence ili.it a-rt.iiii 
things will come alx)iit in the futvire or tliat others will not. 

(Meiiawar 1984) 

In this final chapter, I shall outline some of the practical procediires to be iol- 
lowed in planning, carrying out, and evaiiiatinga reseanli project. Wliile tlie 
size and complexity of projects will vary, from relatively modest, action-ori- 
ented studies carried out by classroom teachers and others involvcti in pro- 
fessional practice through to major investigations by university based 
researchers, all research, 1 believe, can benefit from the considerations con- 
tained in this chapter. The following c|uesti<>ns are de.ilt with: 

- How do I go about developing a research question? 

- What is a literature review, and how do I carry out one? 

- What should a research report contain? 

- What arc some of the pr.ictical problems associated with carrying out 
research, and how might these be dealt with? 



Developing a research question 

A minimum requirement for an activity to Ix' consiilered research is that it 
contain three components: 

1. a question 

2. data 

3. analysis and interpretation. 

The first component, and the key to the others, is tlie fornnilation of a ques- 
tion. It is this initial step which often causes researchers, particularly thcjsc 
who are new to the research process, the most trouble. It is worth speiuliiig 
as much time as is necessary to get the question right, and in this section, we 
shall look at some of the considerations which will facilitate this process. 

Before formulating a question, one needs to determine the genera! to[jie 
area one is interested in. The general area itself can emerge from one's own 
particular interests, from one's experience in the field, from reailing widely, 
or from a combination of these. Table 10. 1 sets out .some of the areas which 
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TAIll K lO.I RHSliAKC H ARKAS AND TOPICS IDENTIFIED BY A GROUP OF 
POSTCiRAOUATF STUDENTS AT MACQUARIE UNIVERSITY 



A rcii 



Pcdapogy 



TcmcIkt 
cdiioitioii 



Sociotiiltiiral 
concerns 



1 r.irnin}', 
Mr;UC(;ics 

Tcstiiin nnil 
assessment 

lanniiap- 
disahility 



Topic 



Tcicliiii}; Korean in secondary schools: motivation and learning 

experiences 

Differences in the discourse of Japanese as a second language 

learners in the playground and the classroom 

The relationship between spt)ken and written language 

Teaching writing for academic purf)oscs 

The written discourse of specialised subjects in the university 

context 

Using tiie genre approach to teach academic writing 
Teacher feedback in the adult RSL classroom 
SI.A in tutored environments 

The role of cultural background in reading comprehension 

Teani teaching: ESLand the mainstream 

A program for community sch(X)l teachers in Papua New 

Guinea 

Professional developinent and program management 

Supervisory feedback in the I KSOL practicuni 

Language use in social context in Japan 

Language and culture in Western Kenya 

Bilingual medical consultations 

Language choice and cultural identity 

Standard versus dialect preferences by Thai speakers in 

Northeastern Thailand 

1 ciriuT independence anil the self-access centre 

Second language strategy preferences of secondary learners 

Testing speaking proficiency 

The predictive validity of the ILL! S" test 

Assessing the language prolicicncy of school-age children 

IVvelopingbilingual/bicultural awareness in primary-age deaf 

students 

The abuse of children with language disability 
Phonological and grammatical performance of students with 
niiltl to nioilcrate sensorineural hearing loss 
Social iiitcr.iction of hearing impaired and hearing children 



'IF.L I S is the International Rnglish Language Testing Service. 
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a group of iny postgraduate students identified. It shows the wide range of 
interests that can einerge from a single group of students, all of whom are 
doing the same course. 

Hatch and Lazaraton (1991) make the excellent practical suggestion that 
graduate students and others who are interested in doing research should 
keep a research journal. Such a journal can be an invaluable resource when it 
comes to defining what interests you and in identifying a research area. They 
suggest: 

Each time you think of a question for which there seems to be no ready answer, 
write the question down. Someone may write or talk about something that is 
fascinating, and you wonder if the same results would obtain with your students, or 
with bilingual children, or with a different genre of text. Write this in your journal. 
Perhaps you take notes as you read articles, observe classes, or listen to lectures. 
Place a star or other symbol at places where you have questions. These ideas will 
then be easy to find and transfer to the journal. Of course, not all of these ideas will 
evolve into research topics. Like a writer's notebook, these bits and pieces of 
research ideas will reformulate themselves almost like magic. Ways to redefine, 
elaborate or reorganize the questions will occur as you reread the entries. (Hatch 
and Lazaraton 1991: 11-12) 

Having identified a general area, and a topic within that area, one begins the 
task of formulating a question. Not all questions are rescarchable. For exam- 
ple, the question 'Should values clarification be taught in primary school?* 
cannot be settled empirically. Of course we could conduct a survey of inter- 
ested community members to find out what they think, but this would not 
provide us with an answer to the question - it would answer a rather different 
one, namely, 'What arc community attitudes towards the teaching of values 
clarification in primary schools?' 

In research, in order to obtain reasonable answers, wc need to ask the right 
sort of questions. The questions need to be: 

1. worth asking in the first place 

2. capable of being answered. 

There are many questions or issues which are eminently capable of being 
researched, but which may not be worth asking. For example, it would be 
technically feasible to determine the number of Spanish interpreters who wear 
designer jeans, or the relationship between the wearing of rubber thongs and 
academic achievement. However, it is highly dubious whether these questions 
are worth asking. Unfortunately, often the questions which are easiest to 
answer are not worth asking. 

On the other hand, there are many questions worth asking which cannot, 
in any practical sense, be answered. So in formulating a research question we 
need to strike a balance between the value of the question and our ability to 
develop a research proposal we arc capable of carrying out. 
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Research questions can come from many different places. Usually they 
result from our reading around in an area that is of interest to us. If we are 
lucky, we may find a piece of published research which we can either take 
one step further or apply to a different context. For example, we may come 
across a study demonstrating that background cultural knowledge has a sig- 
nificant effect on reading comprehension. This may prompt us to ask, 'What 
is the effect of background knowledge on listening comprehension?' Or, we 
may come across research which shows that the order in which German mor- 
phosyntax is acquired is relatively fixed, impervious to instruction, and deter- 
mined by speech processing constraints. This may prompt us to ask whether 
the same constraints exist for English. 

Having identified a general area, such as listening comprehension, or mor- 
phosyntactic development, we need to refine our question so that it is worth- 
while and doable. We may start out with a worthwhile question such as: 'Is 
there an impermeable order of morphosyntactic acquisition for English, 
which can be accounted for in terms of speech processing constraints.^' Such 
a question, however, could occupy a team of postdoctoral researchers for sev- 
eral years. 

Recognising the breadth of the issue, we may decide to chip away at a small 
piece of the puzzle. After thinking and reading around the area, we may 
decide that we will restrict our attention to the acquisition of question forms. 
Our new question might read: 'What is the order of acquisition of question 
forms in English? Can this order of acquisition be accounted for in terms of 
speech processing constraints?' 

At this pKjint there are two things to note. In the first place, the question 
has been derived from the literature, and in the second place, it is theoretically 
motivated. In other words, it is underpinned by a theory of language acqui- 
sition, which is itself based on a broader cognitivist view of learning. 

Our next step is to begin thinking about the data we might collect to 
explore our questions. In order to do this, we need to develop a research pro- 
posal. The proposal needs to take into consideration the issues of validity and 
reliability. Will we collect our data in a laboratory setting, thus guarding 
against threats to internal validity but possibly weakening our ability to 
generalise the results, or will we collect the data in the field? In coming to our 
decision, we might consider Beretta's (1986a) comments on the tension cre- 
ated when we try to deal with threats to the internal and external validity of 
our research. (You will recall Beretta's assertion, reported in Chapter 1, that 
strengthening internal validity by collecting our data through some sort of 
controlled experiment may lead to a situation in which outcomes can not be 
generalised to non-experimental contexts.) 

In addition to internal/external reliability/validity, we need to operation- 
alise the constructs we are working with. In other words, we need to describe 
their characteristics in a way which would enable an outsider to identify them 
if they came across them. If researchers fail to provide 'up front' definitions. 
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then wc need to rc.id Ivtwcen the liiu's. for ex.implf, if a stiiily inviAtig.ui-s 
'listening comprehension', and the dependent variable is a written (. loze test, 
then the default definition of 'listening compri'licnsiun' is 'liu' .ibillty to com- 
plete a written cloze passage'. If we were ro fi[id such a iletiiiition unaccept- 
able, we would be questioning the construct validity oi the study. Construct 
validity has to do with the question: Is the study actiially investigating what 
it is supposed to be investigating? 

In developing a research prohletn or question, Wiersnia ( 1 9S6) suggests that 
the problem be broadly stated in the first instance and then progressively 
refined and restricted through a review of the literature. I le provides the fol- 
lowing examples of how an initial broad area can Iv refornuilated, either as 
a problem statement or question. 



Original Achievement and teaching teclinii]ucs 

Restatement A study of the effects of three tcachitig tecliiiiques on science 

achievement of junior high sch(H)! studcms 
Question Do three different teaching techniques Imve Jiffi ring cffccis on 

science achievement scores of jiniior high school suulcnts? 

Original Bilingual education 

Restatement A study of the nature and cliar.ictcristics of bilingual educ.ilion in 

the elementary schools of (aty A 
Question What are the nature and characteristics of bilingual education as it is 

implemented in tiie elementary schools of City A? 

Original The role of the guidance counselor 

Restatement A survey of ihe practices of the guidance counselors in ilic high 
sch(x)!s of C'ity B 

Questions What pro{X)rtion of guidance counselors' working day is taken up 
with nonguidancc activities? 
What are the major strengths of guidance counselors' practices as 

perceived by the students? 
What are the major weaknesses of guidance counselors' practices as 

perceived by the students? 
What practices are perceived by guidance counselors as most 
effective in advising students about college selection? 

(Wiersnia 1986: 3 1 -.52) 



Seliger and Shohamy (1989) suggest that the preparatory stage of any 
research project should consist of four phases: 

Phase 1: Formulating the general question (this may eitierge from the expe- 
rience and interests of the researcher, other research in language, or sources 
outside second language acquisition) 

Phase!: Focusing the question (here the researcher decides on the importance 
and feasibility of the question) 
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General area: 
Research question(s): 
Key constructs: 
Justification: 
Subjects: 

Procedure and methods: 
Type of data: 
Type of analysis: 
Outcome(s): 
Anticipated problems: 
Possible solutions: 
Resources required; 

l-ignre 1 0.1 A research oullinc to guide in the development of a research 
project 

Phase Deciding on nii objective 

Phase 4: 1-ornuilatiiij; the research plan or objective. 

1 have h)nnd that the creation of a research outHnc at the beginning of the 
project cm greatly facilitate the planning process. The outline in Figure 10. 1 
i,s one used by my graduate students, who add a short statement under each 
heading as rhey develop and refine their project. 

The literature review 

An essential step in any research project is the literature review. The function 
ot the literature review is to provide background information on the research 
question, and ro identify what others have said and/or discovered about the 
(.|ucstion. It may well be that in the course of carrying out the literature 
review, yoii come across a study which answers the very question you arc 
proposing to investigate. The literature review, if carried out systematically, 
will acquaint you with previous work in the field, and should also alert you 
to problems .md potential pitfalls in the chosen area. 

A good way to begin a literature review is to prepare an annotated bibli- 
ography. As the name suggests, an annotated bibliography contains a list of 
relev.int studies relating to the research question or issue. These may range 
from brief research reports to books. Each entry contains a summary or 
abstract of the particular work. The following sample extract from a com- 
mercially published literature review provides some idea of the length and 
derail of rhe annotations. 

KcniK'ily, C:. |4S7. Innovnrioii for a change: teacher development and innovation. 
l-L iJiHinuil, 41, y. 163-170. 
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Author describes a university level ESP project in Tunisia where the aim was to 
further tcicher education. Teachers were involved with materials production. They 
were required to produce a materials blueprint taking into consideration questions 
of approach and design, which make them aware of gaps in their theoretical 
knowledge about language and learning. Kennedy believes that this approach, i.e. 
creating a situation which generates a demand for theory arising from a 
questioning of practice, is the best way to influence deep seated value and belief 
systems and thus to have a lasting effect on teacher behaviour. 

(Dallas 1990) 

Abstracts such as these can be either kept on index cards or entered into a 
word processing or computer database ro be drawn upon in the creation of a 
literature review. A literature review differs from an annotated bibliography 
in that the researcher extracts and synthesises the main points, issues, find- 
ings, and research methods which emerge from a critical review of the read- 
ings, Merriam (1988) suggests that, in carrying out a literature review, it is a 
good idea to differentiate between data-based research and non-data-based 
writings. As the name suggests, data-based literature is based on empirical 
information collected by the researcher. Non-data-bascd writings, on the 
other hand, 'reflect the writer's experiences or opinions and can range from 
the highly theoretical to popular testimonials' (Merriam 1988: 61). 

Wicrsma (1986) provides the following practical advice for carrying out a 
literature review: 

1. Select studies that relate most directly to the problem at hand. 

2. Tie together the results of the studies so that their relevance is clear. Do not 
simply provide a compendium of seemingly unrelated references in paragraph 
form. 

3. When conflicting findings are reported across studies - and this is quite common 
in educational research - carefully examine the variations in the findings and 
possible explanations for them. Ignoring variation and simply averaging effects 
loses information and fails to recognize the complexity of the problem. 

4. Make the case that the research area reviewed is incomplete or requires 
extension. This establishes the need for research in this area. (Note: This docs 
not make the case that the proposed research is going to meet the need or is of 
significance.) 

5. Although information from the literature must be properly referenced, do not 
make the review a series of quotations. 

6. The review should be organized according to the major points relevant to the 
problem. Do not force the review into a chronological organization, for 
example, which may confuse the relevance and continuity among rhe studies 
reviewed. 

7. Give the reader some indication of the relative importance of results from studies 
reviewed. Some results have more bearing on the problem than others, and this 
should be indicated. 

8. Provide closure for the section. Do not terminate with comments from the final 
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study reviewed. Provide a summary and pull together the most imptirtant 
points. 

(pp. .?76-377) 

Numerous resources exist to facilitate the literature review process, and most 
of these can be found in university and college libraries. If one has only a 
vague idea of the general area one wishes to investigate, it is a good idea to 
consult an educational encyclopedia to obtain a general overview of the area 
in question. The library subject catalogue can also provide information on 
relevant books and periodicals. In order to obtain references to journals it is 
best to start with a periodical index such as ERIC (Educational Resources 
Information Centre) or the Reader's Guide to Periodical Literature. The 
ERIC Thesaurus provides descriptors of concepts. Subject indexes such as the 
Current Index to Journals in Education (CIJE) and Resources in Education 
(RIE)\vill give you lists of resources. Finally, there are the specialised abstract- 
ing journals such as Language Teaching, which provide summaries of recently 
published journal articles. 



Implementing the research project 

Most of this book has been devoted to an analysis of different methods of 
collecting and analysing data, and in this section, I should like to summarise 
some of the main points which have been made in greater detail in the body 
of the book. 

In Chapter 1 , we looked at different traditions in research, and I contrasted 
psychometric with interpretive research. While the distinction between these 
traditions is, in many ways, oversimplistic, I argued that the distinction was 
a real one, and that different conceptions of reality and the nature of evidence 
underlay the different traditions. I also took pains to point out that, despite 
the impression which is sometimes conveyed by proponents of the different 
approaches, there is no intrinsic superiority in one rather than the other. In 
selecting a general orientation, it is important to match one's data collection 
methods and methods of analysis to the question one is asking. Some ques- 
tions, particularly those positing a strong causal relationship between vari- 
ables, suggest some form of experimental research design and the use of sta- 
tistical tools to analyse the data and make inferences from one's sample to the 
larger population. Other questions, particularly those concerned with inves- 
tigating behaviour in context, suggest descriptive and interpretive research. 

In addition to ensuring consistency, and to ensure that the data collection 
and analysis are going to answer your questian(s) (and not some other ques- 
tion), there are several other key considerations to be borne in mind during 
the data collection and analysis phases of the research. In particular, one 
needs to ensure that adequate care has been taken over the constructs under- 
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i lying the research and the way in which these have been opcr.ition.iliscd. You 
will recall that this involved defining the constructs and developing opera- 
tional measures of rlie constructs which .irc accessible to the outsitle observer. 

j It is also important, not only during the planning hut also during the imple- 
mentation phases of the project, to be aware of possible threats to the v.ilidity 
and reliability of the research, and ways in which these threats nught be dealt 
with. 

In moving from a general consideration of the research process to the prac- 
ticalities of planning and implementing a research project, it is a good idea to 
try and anticipate the practical problems likely to be encountered, and some 
of the potential solutions to the problems. Problems encountereil by gradu.ue 
students, teachers, and otiiers carrying out action-based researdi include the 
following: 

1. Lack of time (this is a particular problem for those who arc working full- 
time) 

2. Lack of expertise, particularly with critical phases of the rese;ireli, such as 
formulating the research question and determining the appropriate 
research design and statistical tools 

3. Difficulty in identifying subjects 

4. Problems in negotiating access to research sites 

5. Issues of confidentiality 

6. Ethical questions relating to collecting data (these arise when one wants 
to collect oral language without alerting subjects to the fact that they are 

I being observed - recall Labov's 'observer's paradox') 

7. Problems flowing from the growth of the project after initiation 

I 8. Sensitivity of repnarting negative findings, particularly if these relate to 
worksites of individuals with whom one is associated 
9. Preparation of a written report of the research. 

Lack of time is, without doubt, the most frequently reported problem from 
researchers. Walker (1985), writing in the context of collalx)rative, action- 
based research in educational settings, suggests the following strategies for the 
time constraint. 

adapt standard methods 

be flexible 

convince authority of need for more resources 

clarification of task in small group/team discussion before whole staff work 

together 
check on present use of time 
timetable for research and support activities 
try to delegate some of your responsibilities 
use resources of students, university, college, t'tc. 
a prof)er meeting calendar 
use parents 
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term in w liicli rcscnrcli undertaken -niitumn most favoiirnble 

niaii^ij^cnicnt team mulertakcn structured servicing of research, implementing 

.icrioii etc. 
close .school early on one day a term 
second staff 

release staff using INSET (in-service education and training) money 

be judicious in collecting data 

seek co-opcrnrion of colleagues 

tenrn teaching 

diin't transcribe everything 

reappraisal of professional priorities 

(p. 193) 

An iiisiglitful and cl<)t|ucin testimonial to the practical and conceptual diffi- 
culties of doing rcscarcli is provided by Dingwall (1984). She contrasts the 
mcssincss of the* research process with tlic 'hygienic' appearance of ihc 
rcscarcli as it actually appears in written reports and accounts. 

It may be too strong to suggest that there is a 'conspiracy of silence' among 
ac.ulciiiics nlxnit the problems, tlic possibilities, the limitations, and the pressures of 
rcsc.irch practice; bur certainly for most graduate researchers working in 
comparative isolation, it is painful to discover the extent of compromise and 
ambiguity itilierent in their work. (1984: 1) 

Bogdaii and Biklen (1982: 147-154, cited in Mcrriam 1988), although work- 
ing within a more general research field, prcjvidc nine practical suggestions 
for facilitating the research process which arc applicable to language research 
aiul can help to obviate some of the pitfalls of doing research, "riicsc arc set 
out in Table 10.2. 

In a recent survey, a group of graduate students of applied linguistics at 
Maccjuarie University were asked to nominate the problctns they had 
eiicouiitercil in the course of carrying out their research, and the solutions 
iliey liad found to these problems. The results of this survey, which arc set 
out in Table 10.3, illustrate the typical concerns of graduate-level 
investigations. 

Presenting the research 

"Tlierc are many ways of presenting research. The traditional way is in the 
form of jxipcr presentations at conferences and, in written mode, as theses, 
journal articles, or monographs. In the case of research carried out as part of 
a formal tertiary award, there is a requirement that the research be presented 
in writing. 1 have found that many students have a great deal of difficulty 
when it comes to writing up their research. The pnKcss itself is greatly facil- 
itated if one has a clear idea of the audience for whom one is writing. In some 
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TABLE 10. Z IDENTIFYING AND REFINING A RESEARCH AREA 



Area 



Strategy 



1. Research focus 



2. Type of study 



3. Developing questions 



4. Ongoing data collection 



5. Oliserver's comments 



6. Memos 



7. Feedback 



8. Explanation 



You must discipline yourself nor to pursue 
everything ... or else you arc likely to wind up 
with data too diffuse and inappropriate for what 
you want to do. 

You should try to make clear in your own mind . . . 
whether you want to do a full description of a 
setting or whether you are interested in generating 
theory about a particular aspect of it. 

Some researchers bring general questions to a study. 
These arc important bccau.se they give ftKus to data 
collection and help organize it as you proceed. . . . 
We suggest that shortly after you enter the field, 
you a.sscss which questions you brought with you 
are relevant and which ones should be reformulated 
to direct your work. 

In light of what you find when you periodically 
review your fieldnotes, plan to pursue specific leads 
in your next data collection session. 

Write observer's comments as you go to stimulate 
critical thinking about what you see and to become 
more than a 'recording machine'. 

Write memos to yourself ab(5ut what you arc 
learning which can provide a time to reflect on 
issues raised in the setting and how they relate to 
large theoretical, methodological, and substantive 
issues. 

Try out ideas and themes on subjects. While not 
everyone should be asked, and while not all you 
hear may be helpful, key inforinants, under the 
appropriate circumstances, can help advance your 
analysis, especially to fill in the holes of description. 

Experiment with metaphors, analogies, and 
concepts. Nearsightedness plagues most research 

Ask the question, 'What docs this remind me 

of?' Another way to expand analytic horizons is to 
try to raise concrete relations and happenings 
observed in a particular setting to a higher level of 
abstraction. 



Source: Adapted from Bogdan and Biklen (1982: 147-154). 
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TABLK 10.3 PROBLEMS ENCOUNTERED AND SOLUTIONS FOUND BY GRADUATE 
STUDENTS IN CONDUCTING RESEARCH 



Research area Problems encountered 



Solutions 



Identifying a - Narrowing the area sufficiently 
research area - identifying an area in which I 
can get help 

- Ensuring adequate subjects for 
area selected 

- Having several competing areas 

- Coming from a non-teaching 
background 

- Constraints imposed by sponsor 

- Identifying an area that is 
practical 

- Matching needs with interests 

- Being pulled between different 
interests 

Developing a - Refining question so it is doable 
question - Formulating operational 

questions 

- Identifying which competing 
question to do 

- Being too ambitious 

- Avoiding vagueness 

- Deciding if one question is 
sufficient 

- Finding inaccessible articles 
and reports 

review - Identifying the steps involved/ 

how to do it 

- Lack of literature in chosen 
area 

- Having too much literature in 
chosen area 

- Lack of non-English literature 

- Being unfamiliar with library 

- Lack of time 

- Knowing what to exclude/ 
include 

- Covering the cost of 
photocopying 

- Finding relevant studies 

Determining - Lack of time 

data - Lack of expertise in 

collection questionnaire design 

methods - Matching method to question 



Conducting a 
literature 



- Get advice from supervisor 

- Get advice from former 
students 

- Do extensive reading 

- Spend time in field 

- Talk to practitioners 



- Get advice from supervisor 

- Study existing research 

- Go to outside agencies 

• Rank order questions 

• Make preliminary data 
analysis 

' Match questions and 
interests 

Use library reader services 
Get help from supervisor 
Find specialist library 
Learn to use ERIC 
Use interlibrary loan 
Use vacation time to do 
review 

Access other literature 
reviews 



Get help from supervisor 
Get help from 
practitioners 
Replicate published study 



TABLE 10.3 (cont.) 



Research area Problems encountered 



Solutions 



- IX'ciding whether to use 
ethnography or experiment 

- Ensuring rcli.ibility and 
validity 

- Finding enough apjiroprinre 
subjects 

- Negotiating release of data 

- C'ollecting data Ix'fore doing 
research nietii(Kis course 

- IX'teriuiniiig own role within 
research site 

- Ch(X)sing between nuiltiple 
types of relevant data 

- Lack of confidence over 
statistics 

- Dealing with the unexpected 

- Managing data that keep 
changing 

Analysing - Identifying equipment 

lata - Dj)ing statistical analysis 

- Making reliable and valid 
interpretation 

- Drawing conclusions from data 

- Collecting too much data 

- Determining whether to 
quantify qualitative data 

- Lack of time 

- Being objective 

- 'Seeing' what is there 

- Lack of confidence 

Drawing - Identifying population to 

conclusions which findings apply 

- Coming to definite conclusions 

- Interpreting statistical data 

- Knowing how far to go 

- Being relevant 

- Overgeneralising from data 

Writing up - Determining appropriate 
research structure 

- Lack of time 

- Being unable to type 

- Covering the cost of binding 



Consult statistician 



Apply for a gr.inl 

Have supiTvisor coili' pari 

of data 

Learn to use computer 



Dim't draw ilctiuite 
conclusions 

Get help from supervisor 
Take care not to 
generalise 



Study other dissertations 
Take annual leave 
Find lxK)k on how to 
write researcli rejiort 
Consult supervisor 



(continued) 
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TAIIL.F. 10.3 (cotlt.) 




Research area I nMems encoutttered 


Solutions 


- Meeting requirements of thesis 


- Do a small amount at a 


writing 


time 


- Knowing where to stnrt 


- Have fellow students 


- Making rcpurt logicnl and 


check draft 


relevant 




- Identifying overall layout 




- l-'iniliiig style appropriate to 




both linguistics and 




socioiinguistics 




- Being concise 




~ Being a non-native speaker 





cases there may Ix; more than one audience - for example, one's supervisor 
and one's collcagucs/pccrs - in which case more than one version of a ref>ort 
may he called for. 

The writing process can also be greatly facilitated if one works out a struc- 
ture for the report before beginning. Figure 10.2 illustrates the structure of 
one comprehensive research report. It can be seen that the report falls natu- 
rally into several sections. It begins with an introduction, purpose, and ratio- 
nale, including an indication of the limitations and delimitations of the 
research. This is followed hy several chapters summarising the literature 
review. The study itself is then described, and the results are presented and 
discussed. The report concludes with a discussion of the theoretical and prac- 
tical implications of the research. 

While research reports will vary, 1 have included Figure 10.2 as a model 
Ix'cause it contains all of the essential elements of a good report. While it has 
licen taken from a psychometric investigation, it can also be used for intcr- 
(jretive research. Once one has identified one's research area, identified the 
rcsc.uch question, and Ix-guii the literature review, it is a good idea to create 
a word processing file with the heading and subheadings one wishes to use. 
As the research project proceeds, data can then be inserted into the report 
uiulcr the appropriate heading, and by the end of the project, you will have 
a draft report. Admittedly, this will need editing, but a great deal of the work 
of writing up the report will already be done. 

Shaw (1983), cited in Chaudron (1988), provides a summary of the staging 
and sequencing of a research project (a doctoral investigation of the language 
of engineering professors). The process distinguishes between 'on-site' and 
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Title; Discourse processing by first language, second phase' and second language 
learners 

List of tables 
List of figures 
Summary 
Acknowledgments 
Chapter 1 Introduction 

The problem 

Purpose of the study 

Significance of the study 

Background to the study 

The aims and justification of the study 

Limitations and deliminations 

What this study proposes 
Chapter 2 Literature review; the reading process 
Chapter 3 Literature review; cohesion and coherence 
Chapter 4 Literature review: the assessment of discourse processing 
Chapter 5 Literature review: comparing first and second language learning 
Chapter 6 Experimental design 

The research questions 

Dermitions of terms 

The research hypotheses 

Subjects used In the study 

Test materials 

Scoring procedures 

Design and administration 

Statistical analysis 

Summary 

Chapter 7 Presentation and discussion of results 
Chapter 8 Theoretical and pedagogical implications 
Chapter 9 Summary and conclusions 

Major aims 

Minor aims 

An evaluation 

Further research 

References 

Appendix Test materials 

"Second phase' learners are learners who have been in the target language 
community long enough to acquire oral fluency and general communication skills, 
but who may have problems with the academic language encountered in school. 



Figure 10.2 The structure of a research report: an example 
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TABLE 10.4 SEQUENCE OF OUTCOMES AND ACTIVITIES IN RESEARCH 



Stdge in Research actiinty 

research 

sequence Research outcome Off-site On-site 



I 


Conditions and 
limits of study 
established; sites 
selected 


Literature reviewed 


Pilot study (ethnography 
and audio recording) 


II 






Longitudinal 
ethnography begins 


III 


Oass for 


Initial professor 


Cross-sectional 




audiotaping 


interviews 


ethnography begins 




selected 






IV 






Audio-recording 
Videotape recording 


V 


'General picture' 


Transcription 


Cross-sectional 




emerges 


begins 

Professors debriefed 


ethnography ends 


VI 


Hyi)<)thescs 


Fieidnotes 


Longitudinal 




generated from 


completed and 


ethnography ends 




transcript 


reviewed 






review 






VII 


Data matched 
against 
hypotheses 
generated in Vf 


Transcripts complete 
Transcripts reviewed 
and analysed 




VIII 


'General model' 
emerges 


Final data analysis 
(integrated analysis) 





'off-site' research activity, and illustrates the range of data and data collection 

methods, as well as the interaction between data collection, hypothesis for- 
ination, and data analysis. Table 10.4 summarises the activities and outcomes 
in a research project. 

Conclusion 

In this chapter, I have related the key issues and concerns from the book as a 
whole to the practical concerns of carrying out research. In Table 10.5, these 
issues and concerns are formulated as a set of questions to be used as a guide 
in the design of a research project. 
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TABLE 10.5 CHECKLIST OF QUESTIONS TO CUIOE THE OESICIN OF A HI SFAKt 11 
PROJECT 



Area 



Questions 



Question 



Design 
Method 



Analysis 



Presentation 
Results 



Is the question worth investigating.' 
Is the question feasible? 

£)oes my research question imply a strong causal relationship 

between two or more vnri.ibics? 
What are the constructs underlying my question? 
How are these to be operationaliscd? 

Does the question suggest an experimental or non-ex[x-riniental 

design? 

What methods are available for investigating the question? 
Which of these are feasible, given available resources and ex|x'rtise? 
Is it possible to utilise more than one data collection method? 
Given my chosen data collection nicrh<Kls, what threats are there to 

the internal and external reliability of the study? 
Given my chosen data collection methods, what threats are there to 

the internal and external validity of the study? 

IXx's my research entail statistical or interpretive analysis or Ixuli? 
IX) I have the necessary skills to carry out tlic siatisiital .uialysis? 
Is it necessary or desirable for me to quantify qualitative data, and 
if so, what means suggest themselves? 

How can the research best be presented? 

What are the outcomes of the research? 

Does the investigation answer the question 1 originally asked? 

Does it answer other questions? 

Are my results consistent with the findings of similar siiulies? 
Are there any contradictory findings? I low can these Ix- accoiiiiieil 
for? 

What additional questions and suggestions for further research are 
thrown up by the research? 



Questions and tasks 

1. Prepare a detailed research plan for a project of your choice. Include infor- 
mation on the following: 

L Background 
General area: 
Title: 

The problem, question, or issue: 
Purpose of the study: 
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Significance of the study: 

Background to the study: 

The aims and justification of the study: 

I. imitations and dcliminations: 

What this study proposes: 
11. A statement relating to the literature review, e.g., resources, journals to 

be consulted 
ill. Experimental design 

The research questions: 

Definitions of terms: 

The research hypotheses: 

Subjects used in the study: 

Data collection methods: 

Data analysis procedures: 
IV. Statement on how results will be presented 

2. Evaluate the proposal against the set of questions in Table 10.4. (It may 
not he ptxssible to provide a response to every question.) 

3. What problems do you anticipate or have you experienced in planning 
and implementing research? Can you think of possible solutions? 

Further reading 

Seliger and Shohamy (1989) contains a number of practical suggestions on 
doing research. An excellent publication which deals at length with the plan- 
ning, implementation, and reporting of qualitative research is Merriam 
(1988). Additional useful suggestions can be found in Nunan (1989), Walker 
(1985), and Wiersma (1986). 
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action research A form of self-reflective inquiry carried out by practitio- 
ners, aimed at solving problems, improving practice, or enhancing under- 
standing. It is often collaborative. 

analysis of covariance (ANCOVA) A more sophisticated version of 
ANOVA (see below) which enables the researcher to statistically control 
for the effect on the dependent variable of variables other than the inde- 
jDendent variable. 

analysis of variance (ANOVA) A statistical procedure for testing the dif- 
ference between two or more means. It is used for estimating the probabil- 
ity that the means have been drawn from the same or different populations. 

applied linguistics A broad field of inquiry concerned with the study of 
language use, language acquisition/learning, and language disability util- 
ising models and concepts from a range of disciplines including theoretical 
linguistics, anthropology, education, sociology, and psychology, it has 
many applications, including language p>edagogy, sp>eech pathology, deaf- 
ness education, translation, lexicography, computational linguistics, and 
stylistics. 

case study The investigation of the way a single instance or phenomenon 
functions in context. In applied linguistics, it usually involves the investi- 
gation of the language behaviour of a single individual or limited number 
of individuals over a p)ericKl of time. 

central tendency The tendency of a set of scores to cluster around a par- 
ticular value. The usual measures of central tendency are the mean, 
median, and the mode. 

chi-square (x^) A statistical procedure for comparing the frequencies of 
two or more samples. 

construct A psychological attribute such as intelligence, aptitude, or moti- 
vation which is created to account for observable behaviour. 

correlation A set of statistical procedures for testing the strength of asso- 
ciation between sets of scores. 

deductivism The testing of a theory through the collection of data. Dcduc- 
tivism contrasts with inductivism, where theory is derived from data. 

diary In language education, a first person account of the experience of lan- 
guage learning or reaching. 
di5f>er5ion TTie tendency for a set of scores to spread out or depart from 
the average or 'r>'pica/' values in the scr of scores. Dispersion is usually mea- 
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sured through the range, the mean deviation, the variance, and the stan- 
dard deviation of the scores. 

distribution The representation of a set of scores according to their fre- 
quency of occurrence (see, for example, Figure 2.1). 

elicitation A range of procedures for obtaining sp>eech samples and other 
data from subjects. Such procedures may range from the administration of 
standardised rests through to questionnaires and interviews. 

ethnographic techniques Techniques such as participant observation, 
non-participant observation, interviews, diaries, and journals for docu- 
menting sociocultural aspects of behaviour in the natural settings in which 
those behaviours occur. 

ethnography A non-manipulative study of the cultural characteristics of a 
group in real-world rather than laboratory settings, utilising ethnographic 
techniques and providing a sociocultural interpretation of the research 
data. 

experiment A procedure for testing an hypothesis by setting up a situation 
in which the strength of the relationship between variables can be tested. 
A true experiment consists of control and experiment groups to which sub- 
jects have been randomly assigned, and in which all subjects are tested 
before and after the intervention or treatment under investigation has been 
administered to the exjjeriment group. A pre-exp)eriment may have pre- 
and posttreatment tests, but lacks a control group. A quasi-experiment has 
both pre- and posttests, and experiment and control groups, but no random 
assignment of subjects. 

factor analysis A number of techniques for studying the correlations 
between a number of variables simultaneously in order to identify patterns 
of relationships among the variables. 

frequency table A table showing the number of times different scores 
occur. 

grounded theory The practice of deriving theory from data rather than 
collecting data with the aim of supporting or refuting a theory. The term 
refers to the fact that theory is grounded in descriptive data from real life 
situations. 

hypothesis A formal statement about an expected relationship between 
two or more variables which can be tested through an experiment. For 
example: 'Field-independent learners will learn grammar more effectively 
through a deductive approach than through an inductive approach'. 

inductivism The development of theories and principles from data col- 
lected through observation. 

inferential statistics Statistics designed to enable the researcher to make 
generalisations about a population from data derived from a sample. 

interaction analysis A family of procedures for analysing and interpreting 
recorded speech data. 
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interpretive research Research based on discursive rather than statistical 
analysis. 

interview I he elicitation ol data by one person Iroin aiioilicr ihrough jx r- 
son-to-person encounters. 

introspection The process of observing and reflecting on one's rlioiiglus, 
feelings, motives, reasoning processes, and mental states with a view to 
determining the ways in which these processes and states ileteriniiie or 
influence behaviour. 

literature review A written summary and critique of research relating to 
a particular issue or question. 

mean (X) The average of a set of scores, obtained by adding the .scores 
together and dividing by the total luiinlxT of scores. 

median That value of a set of scores which has the same lunnbcr ol obser- 
vations above and below it when the observations are ranked from highest 
to lowest. 

mode The value which occurs most frequently in .1 set of scores, 
naturalistic-ecological hypothesis The Ix'lief that the context in which 

behaviour occurs will have a significant effect on that Ix-haviour. 
population All cases, situations, or individuals who share one or more 

characteristics, 
pre-experiment See experiment. 

program evaluation The systematic collection and interpretation of data 

about one or more aspects of a program with a view to improving practice, 
protocol A written record of a subject's data, usually obtained through 

some form of elicitation. 
psychometric research Research carried out by the collection of data 

through an experiment, and the analysis of that data through the use of 

inferential statistics, 
qualitative data Data which are recorded in non-numerical form, such as 

transcripts of classroom interactions, 
qualitative-phenomenologicai hypothesis The belief that there is no 

objective reality that is separate from the observer, 
quantitative data Data which are recorded in numerical form, 
quasi-experiment See experiment. 

questionnaire An instrument for the collection of data, usually in writ ten 
form, consisting of open and/or closed questions and other probes reqm'r- 
ing a response from subjects. 

range The difference between the highest and lowest values in a .set of 
scores. 

reliability The extent to which (a) an independent researcher, on analysing 
one's data, would reach the same conclusions and (b) a repiicarioii of one's 
study would yield similar results. Interrtal reliability refers to the consis- 
tency of the results obtained from a piece of research. Externul reliahility 
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refers to the extent to which independent researchers can reproduce 
n study mid obtain results similar to those obtained in the original 
study. 

research A systematic process of inquiry consisting of three elements or 
components: (1) a question, problem, or hypothesis, (2) data, and (3) anal- 
ysis and interpretation of data. 

sample A subset of individuals or cases from within a fxjpulation. 

second language acquisition (SLA) The processes through which individ- 
uals develop skills in a second or foreign language in tutored or untutored 
environments. 

single case research The investigation of a single subject in which the 
experimenter intervenes and tests the effect of this intervention on the sub- 
ject's Ix^hnviour. 

standard deviation (SD) A measure of the dispersion of a set of scores from 
the mean of the scores. It is calculated by obtaining the square root of the 
vnri.nncc of a set of scores. 

standard error (SE) The standard deviation of sample means. For a given 
sninplc, it can Ix: calculated by dividing the standard deviation of the sam- 
ple by the square root of the number of observations in the sample. 

statistical inference The process of making judgments about the charac- 
teristics of an entire population based on data from a subset or sample of 
that population. 

statistics Sets of mathematical procedures for collecting, classifying, and 
analysing quantitative data. 

stimulated recall A technique in which the researcher records behaviour, 
usually on video- or audiotape, and then gets the subject to comment on 
the Ix'haviour, using the recording as an aid to memory. 

survey The collection of data (usually related to attitudes, beliefs, or inten- 
tions) from subjects without attempting to manipulate the phenomena/ 
vari.iblcs under investigation. 

think aloud A data collection technique in which subjects verbalise their 
thought processes as they complete a task or solve a problem. 

true experiment Sec experiment. 

f-tcst A statistical procedure for testing the difference between two or more 

means. It is used for estimating the probability that the means have been 

drawn from the .same or different populations, 
validity The extent to which one has really observed what one set out to 

t)bserve, and the extent to which one can generalise one's findings from the 

subjects and situations to other subjects and situations, 
variable A property or characteristic which may differ from individual to 

individual or from group to group. A great deal of research is carried out 

in order to identify or test the strength of relationships between variables. 

When one variable influences or affects a second variable, the first variaNf 
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is called an independent variable, and the second is called a dependent 

variable. 

variance A measureof dispersion, calculated for a set of scores by subtract- 
ing each score from the mean, squaring the resulting values, adding these 
together, and dividing by the remainder of the number of scores minus 1. 
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